University of Toronto University of Toronto Department of Computer Science Knowledge Elicitation Techniques in RE Lecture 4 Requirements Elicitation II Last Last Week Week Elicitation Elicitation I I Traditional Traditionalapproaches approaches Interviews Questionnaires Interviews Questionnaires Scenarios Scenarios Goals Goalsand andUse Cases Use Cases But KE is hard Protocol Analysis Protocol Analysis Using UsingMultiple MultipleExperts Experts Machine Learning Machine Learning epistemological inadequacy expressiveness vs acquirability Expert Bias 1 Department of Computer Science Why is KE so hard 2000 2003 Steve Easterbrook 2 University of Toronto Department of Computer Science Expressiveness vs Acquireability Experts are not used to describing what they do Three stage model of learning Form Filling Interfaces 1 cognitive verbal rehearsal of tasks 2 associative reinforcement through repetition verbal mediation disappears 3 autonomous compiled no conscious awareness of performance Ideal Representation Procedural and declarative are different mechanisms Declarative knowledge becomes procedural with repeated application experts lose awareness of what they know and cannot introspect reliably Experts have little or no introspective access to higher order cognitive processes Acquiribility Rule Induction Interfaces Representational Problems Spreadsheet Programs Experts don t have the language to describe their knowledge No spoken language offers the necessary precision Knowledge Engineer and Expert must work together to create a suitable language Expert System Shells Different knowledge representations are good for different things Epistemological adequacy does the formalism express expert s knowledge well Eliciting Elicitingperformance performanceknowledge knowledge Representational Problem 2000 2003 Steve Easterbrook Card Sorting Card Sorting Laddering Laddering Proximity Scaling Techniques Proximity Scaling Techniques Automated AutomatedTechniques Techniques Brittleness Assumption of rationality Next Next Week Week Example Example Techniques Techniques Eliciting Elicitingdomain domainknowledge knowledge Delphi Technique Delphi Technique Focus Groups Focus Groups Repertory Grids Repertory Grids Separation of domain knowledge from performance knowledge Modeling problems Modeling Modelingand andAnalysis Analysis I I Modeling ModelingGoals Goals Modeling Organisations Modeling Organisations Modeling ModelingNon Functional Non FunctionalReqs Reqs Background Knowledge elicitation is concerned with discovering expert knowledge Grew out of Expert Systems work in the 80 s Originally focussed on deriving expert s rules for Rule based Systems More recently focussed on problem solving methods This This Week Week Elicitation Elicitation II II Cognitive Cognitive approaches approaches Contextual Contextual approaches approaches Ethnography Ethnography as as an an RE RE technique technique University of Toronto Department of Computer Science Logic Program Environments Brittleness Knowledge is created not extracted Knowledge models are abstractions of reality and hence are unavoidably selective Brittleness caused by the simplifying assumptions instead of adding more knowledge a better more comprehensive model is needed 2000 2003 Steve Easterbrook Expressiveness 3 2000 2003 Steve Easterbrook 4 1 The Knowledge Level knowledge modelling as Observe behaviour of an agent as black box Construct two models Symbol Level descriptions for mechanising behaviour Knowledge Level descriptions of the agent s knowledge of the world Two step Environment Behaviour It acts as if it has some knowledge about its environment which it uses rationally It takes actions to achieve ascribed goals rationality Domain model a systematic way of talking about a domain with a coherent ontology Task model models goals what it means to achieve a goal and how goals are related Problem solving method a way of relating task and domain models to accomplish goals Think aloud vs retrospective protocols Advantages Direct verbalisation of cognitive activities Embedded in the work context Good at revealing interaction problems with existing systems Disadvantages Essentially based on introspection hence unreliable No social dimension Symbol level model First creates a task specific model from the KL model based on features of the task step 1 pairwise proximity assessment among domain elements step 2 automated analysis to build multi dimensional space to classify the objects Advantages help to elicit mental models where complex multivariate data is concerned good for eliciting tacit knowledge Behaviour Disadvantages Requires an agreed on set of objects Only models classification knowledge no performance knowledge Task features 2000 2003 Steve Easterbrook 5 University of Toronto Proximity Scaling Techniques Given some domain objects derive a set of dimensions for classifying them Knowledge Problem solving method Protocol Analysis based on vocalising behaviour Observer modeller Agent applies its knowledge in two stages Hence we actually need 3 models observe Agent 2000 2003 Steve Easterbrook Card Sorting For a given set of domain objects written on cards Expert sorts the cards into groups then says what the criterion was for sorting and what the groups were Advantages simple amenable to automation elicits classification knowledge Problems suitable entities need to be identified with suitable semantic spread across domain No performance knowledge 6 Source Adapted from Hudlicka 1996 University of Toronto Department of Computer Science more KE techniques Department of Computer Science Knowledge Elicitation Techniques Knowledge level model mechanise View University of Toronto Department of Computer Science rationalise University of Toronto Department of Computer Science KA from Multiple Experts Laddering Delphi technique Used where contact between experts is difficult Uses a set of probes types of question to acquire structure and content of stakeholders knowledge Interview the expert Use questions to move up and down a conceptual hierarchy Each expert submits their judgement All judgements are circulated anonymously to all experts Each expert then submits a revised judgement Iterate until judgements converge Focus Groups A technique derived from marketing Assemble experts together and discuss the problem Discussion may be structured e g debate or unstructured Advantages deals with hierarchical knowledge including polyhierarchies e g goal trees