Department of Computer Science University of Toronto CSC2130 Empirical Research Methods for Software Engineering Steve Easterbrook sme cs toronto edu www cs toronto edu sme CSC2130 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 1 Department of Computer Science University of Toronto Course Goals Prepare students for advanced research in SE Learn how to plan conduct and report on empirical investigations Understand the key steps of a research project formulating research questions theory building data analysis using both qualitative and quantitative methods building evidence assessing validity and publishing Motivate the need for an empirical basis for SE Cover all principal empirical methods applicable to SE controlled experiment case studies surveys archival analysis action research ethnographies Relate these methods to relevant metatheories in the philosophy and sociology of science 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 2 1 Department of Computer Science University of Toronto Intended Audience This is an advanced software engineering course assumes a strong grasp of the key ideas of software engineering and the common methods used in software practice Focus how do software developers work how do new tools and techniques affect their ability to construct high quality software efficiently qualitative and quantitative techniques from behavioural sciences The course is aimed at students who plan to conduct SE research that demands some form of empirical validation wish to establish an empirical basis for an existing SE research programme wish to apply these techniques in related fields e g HCI Cog Sci Note we will not cover the kinds of experimental techniques used in CS systems areas 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 3 Department of Computer Science University of Toronto Format Seminars 1 three hour seminar per week Mix of discussion lecture student presentations Readings Major component is discussion of weekly readings Please read the set papers before the seminar Assessment 10 Class Participation 20 Oral Presentation critique a published empirical study 70 Written paper design an empirical study for a SE research question 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 4 2 Department of Computer Science University of Toronto Course Outline 1 Introduction Orientation 2 What is Science Philosophy of Science Sociology of Science Metatheories 3 What is software engineering Engineering Design Disciplinary Analogies for SE Evidence based software engineering 4 Basics of Doing Research Finding good research questions Theory building Research Design Ethics Evidence and Measurement Sampling Peer Review Process 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 5 Department of Computer Science University of Toronto Course Outline cont 5 Experiments 10 Controlled Experiments Quasi experiments Replication 6 Case Studies Action Research Pilot Studies Benchmarking 11 Single and Multi case Longitudinal Case Studies Approaches to Data Collection 7 Reading Week No seminar 8 Histories and Simulations Artifact Analysis Archival Analysis and Post mortems Simulation Techniques 9 Survey and Observation Surveys Focus Groups Field Studies Ethnographies Interventions Analysis Methods Qualitative Quantitative and Mixed approaches Statistical Analysis Grounded Theory 12 Generalisation and Validity Threats to Validity Power and Reliability Replication 13 Reporting and Publishing Displaying data Writing up results Where to publish 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 6 3 Department of Computer Science University of Toronto A Scientific Approach No single official scientific method http dharma haven org science myth of scientific method htm However there are commonalities Observation Theory World Validation 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license University of Toronto 7 Department of Computer Science High School Science Version 1 Observe some aspect of the universe 2 Invent a tentative description called a hypothesis that is consistent with what you have observed 3 Use the hypothesis to make predictions 4 Test those predictions by experiments or further observations and modify the hypothesis in the light of your results 5 Repeat steps 3 and 4 until there are no discrepancies between theory and experiment and or observation 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 8 4 Department of Computer Science University of Toronto Inquiry Cycle Prior Knowledge Initial Hypothesis Observe what is wrong with the current theory Theorize Experiment refine create a better theory manipulate the variables Design Design empirical tests of the theory 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license University of Toronto 9 Department of Computer Science Some Characteristics of Science Explanations are based on observations A way of thinking Relationships are perceptible in a way that has to make sense given accepted truths Creativity is as important as in art Hypotheses experimental designs Search for elegance simplicity 2006 7 Steve Easterbrook This presentation is available free for non commercial use with attribution under a creative commons license 10 5 Department of Computer Science University of Toronto Some Definitions A model is an abstract representation of a phenomenon or set of related phenomena Some details included others excluded A theory is a set of statements that explain a set of phenomena Serves to explain and predict A hypothesis is a testable statement derived from a theory A hypothesis is not a theory In software engineering there are few capital T theories Many small t theories philosophers call these folk theories 2006 7 Steve Easterbrook This presentation is available free for non commercial use with