Experimentation in Software Engineering:Reading Studies The Software Engineering DisciplineSoftware techniques, methods, models. etc. need to bevalidated via experimentationrefined and tailored to the application environmentlogically or physically integratedbe easily transferred into practiceThere is a need to understand the relationships between process and productlearn and evolve our knowledge based upon experienceWe need an experimental, evolutionary software development frameworkthat deals with the symbiotic relationship between research and developmentso that learning can take place in a practical wayExperimentation can take many formsResearch ParadigmsNeed research to establish a scientific and engineering basis for software engineering Required research methods involve the need to build, analyze, and evaluate models of the software processes and products various aspects of the environment, e.g the people, the organizationand the interactions of these models. The goal is to develop the conceptual scientific foundations of software engineering upon which future researchers can build. This is often a process of discovering and validating small but important concepts that can be applied in many different ways and that can be used to build more complex and advanced ideas rather than merely providing a tool or methodology without experimental validation of its underlying assumptions or careful analysis and verification of its propertiesResearch ParadigmsDefinitionsA fact is information obtained through direct observationA hypothesis is an educated guess that precedes an experimentAn experiment is -- a test, trial or tentative procedure policy;-- an act or operation for the purpose of discovering something unknown or of testing a principle, supposition, etc.; -- an operation carried out under controlled conditions in order to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known lawResearch ParadigmsDefinitionsA theory is a possible explanation based upon many facts and reasonA law is a description/observation of behavior used for prediction based upon facts and reasonA model is a simplified representation of a system or phenomenon with any hypotheses required to describe the system or explain the phenomenon, often mathematically. A model can be a theory or a lawA paradigm is conceptual filter that determines how we perceive/interpretA truth is what really isResearch ParadigmsResearch methods from other disciplines include various forms ofexperimental or analytic paradigmsExperimental paradigms require an experimental design, observation, data collection and validation on the process or product being studiedThe Scientific Method:observe the world,propose a model or a theory of behavior,measure and analyze,validate hypotheses of the model or theory,and, if possible, repeat the procedure.Research ParadigmsThe Scientific Method is an inductiveparadigm which can be used to:understand the software process, product, people, environmentextract models from the world that explain underlying phenomena, andevaluate if the model is representative of the phenomenon observedExample: an attempt to understand the way software is developed by an organization to see if their process model can be abstracted or a tool can be built to automate the processThere are two variations of the inductive paradigm which we will call theengineering methodempirical methodResearch ParadigmsThe Engineering Method:observe existing solutions,propose better solutions,build/develop,measure and analyze, andrepeat the process until no more improvements appear possible.This version of the paradigm is an evolutionaryimprovement oriented approachassumes models existmodifies model to improve the thing being studiedExample: study improvements to methods or demonstrate that a tool performs better than its predecessor relative to certain characteristicsResearch ParadigmsThe Empirical Method:propose a model,develop statistical/qualitative methods,apply to case studies,measure and analyze,validate the model and repeat the procedure.This version of the paradigm isa revolutionaryimprovement oriented approachproposes a new modelstudies effects of process or product suggested by the new modelExample: proposal of a new method or tool and validation that the model or tool is an advance over current models or toolsResearch ParadigmsThere must be a rationale for collecting data. Experiments must be designed to acquire information useful for the building of a suitable description (model or theory) of the systems under study. It is an approach to model/theory/law building. Experimentation alone is of no value if there is no underlying framework or context where experimental results can be interpreted. Other issues involved in these inductive, experimental methods include - the types of experimental design appropriate for different environments, - whether the experiment is exploratory or confirmatory, - the validity of the data collected, - the cost of the experiment, - the problems of reproducibility, etc.Research ParadigmsAn analytic paradigm is:The Mathematical Method:propose a formal theory or set of axioms,develop a theory,derive results,andif possible compare with empirical observations.This is a deductiveanalytical model whichdoes not require experimental design in the statistical sense, but provides a framework fordeveloping models and understanding their boundariesbased upon manipulation of model itselfExample: the treatment of programs as mathematical objects and the analysis of the mathematical object or its relationship to the programResearch ParadigmsThese paradigms serve as a basis for distinguishing research activities from development activitiesIf one of these paradigms is not being used in some form, the study is most likely not a research projectMany projects that claim to be research are simply developments, e.g., building a system or tool alone is development, not researchResearch involves gaining understanding about how and why a certain type of tool might be useful, and by validating that a tool has certain properties or effects by carefully designing an experiment to measure the properties or to compare it with alternativesThe scientific method can be used to understand the effects of a particular tool in some environment and to validate hypotheses about how software development can best be accomplishedThe Experimental Discipline How do we combine experiments?There