University of Toronto Thin spread of domain knowledge The knowledge might be distributed across many sources Basics of elicitation It is rarely available in an explicit form I e not written down There will be conflicts between knowledge from different sources Why info collection is hard Remember the principle of complementarity Dealing with Bias Department of Computer Science Difficulties of Elicitation Lecture 9 Eliciting Requirements University of Toronto Department of Computer Science A large collection of elicitation techniques Tacit knowledge The say do problem People find it hard to describe knowledge they regularly use Background Reading Hard data collection Interviews Limited Observability The problem owners might be too busy coping with the current system Questionnaires Presence of an observer may change the problem Group Techniques E g Probe Effect Hawthorne Effect Participant Observation Ethnomethodology Bias People may not be free to tell you what you need to know Knowledge Elicitation Techniques People may not want to tell you what you need to know 2 Easterbrook 2004 University of Toronto Department of Computer Science The outcome will affect them so they may try to influence you hidden agendas Easterbrook 2004 University of Toronto Example What Loan approval department in a large bank The analyst is trying to elicit the rules and procedures for approving a loan can there ever be no bias Why this might be difficult All views of reality are filtered All decision making is based partly on personal values Implicit knowledge There is no document in which the rules for approving loans are written down Types Conflicting information Different bank staff have different ideas about what the rules are of bias Motivational bias Say do problem expert makes accommodations to please the interviewer or some other audience The loan approval process described to you by the loan approval officers is quite different from your observations of what they actually do Observational bias Probe effect Limitations on our ability to accurately observe the world The loan approval process used by the officers while you are observing is different from the one they normally use Cognitive bias Bias Mistakes in use of statistics estimation memory etc The loan approval officers fear that your job is to computerize their jobs out of existence so they are deliberately emphasizing the need for case by case discretion to convince you it has to be done by a human Easterbrook 2004 is bias Bias only exists in relation to some reference point Notational bias Terms used to describe a problem may affect our understanding of it 4 Easterbrook 2004 3 Department of Computer Science Bias Examples Examples of of Bias Bias Social Social pressure pressure response responsetotoverbal verbaland andnon verbal non verbalcues cuesfrom from interviewer interviewer Group Group think think response responsetotoreactions reactionsof ofother otherexperts experts Impression Impression management management response responsetotoimagined imaginedreactions reactionsof ofmanagers managers clients clients Wishful Wishful thinking thinking response responsetotohopes hopesor orpossible possiblegains gains Appropriation Appropriation Selective Selectiveinterpretation interpretationto tosupport supportcurrent currentbeliefs beliefs Misrepresentation Misrepresentation expert expertcannot cannotaccurately accuratelyfit fitaaresponse responseinto intothe the requested response mode requested response mode Anchoring Anchoring contradictory contradictorydata dataignored ignoredonce onceinitial initialsolution solutionisis available available Inconsistency Inconsistency assumptions assumptionsmade madeearlier earlierare areforgotten forgotten Availability Availability some somedata dataare areeasier easiertotorecall recallthan thanothers others Underestimation Underestimation of of uncertainty uncertainty tendency tendencytotounderestimate underestimateby byaafactor factorof of22or or3 3 5 1 University of Toronto University of Toronto Department of Computer Science Elicitation Techniques Traditional techniques Discourse Analysis Interviews Conversation Analysis Speech Act Analysis Open ended Structured Soft Systems Analysis Meetings Helps to prepare for other types of fact finding Cognitive techniques Focus Groups Brainstorming JAD RAD workshops Prototyping Participatory Design Advantages Helps the analyst to get an understanding of the organization before meeting the people who work there Sociotechnical Methods Surveys Questionnaires Sources of information company reports organization charts policy manuals job descriptions reports documentation of existing systems etc Participant Observation Enthnomethodology Analyzing hard data Collaborative techniques Ethnographic techniques Reading existing documents Background Reading Contextual social approaches Introspection Department of Computer Science e g by being aware of the business objectives of the organization Task analysis may provide detailed requirements for the current system Protocol analysis Knowledge Acquisition Techniques Card Sorting Laddering Repertory Grids Proximity Scaling Techniques Disadvantages written documents often do not match up to reality Can be long winded with much irrelevant detail Appropriate for Whenever you not familiar with the organization being investigated 6 Easterbrook 2004 University of Toronto Department of Computer Science Hard Data and Sampling Hard data includes facts and figures University of Toronto Example of hard data 7 Department of Computer Science Questions Forms Invoices financial information Reports used for decision making Survey results marketing data Easterbrook 2004 What does this data tell you Sampling What would you do with this data Sampling used to select representative set from a population Purposive Sampling choose the parts you think are relevant without worrying about statistical issues Simple Random Sampling choose every kth element Stratified Random Sampling identify strata and sample each Clustered Random Sampling choose a representative subpopulation and sample it Sample Size is important balance between cost of data collection analysis and required significance Process Easterbrook 2004 Decide what data should be collected e g banking transactions Determine the population e g all transactions at 5 branches over one week Choose type of sample e g simple random sampling Choose sample size e g every 20th transaction 8 Easterbrook
View Full Document
Unlocking...