University of Toronto Thin spread of domain knowledge The knowledge might be distributed across many sources Basics of elicitation It is rarely available in an explicit form I e not written down There will be conflicts between knowledge from different sources Why info collection is hard Remember the principle of complementarity Dealing with Bias Department of Computer Science Difficulties of Elicitation Lecture 9 Eliciting Requirements University of Toronto Department of Computer Science A large collection of elicitation techniques Tacit knowledge The say do problem People find it hard to describe knowledge they regularly use Background Reading Hard data collection Interviews Limited Observability The problem owners might be too busy coping with the current system Questionnaires Presence of an observer may change the problem Group Techniques E g Probe Effect Hawthorne Effect Participant Observation Ethnomethodology Bias People may not be free to tell you what you need to know Knowledge Elicitation Techniques People may not want to tell you what you need to know 2 Easterbrook 2004 University of Toronto Department of Computer Science The outcome will affect them so they may try to influence you hidden agendas Easterbrook 2004 University of Toronto Example What Loan approval department in a large bank The analyst is trying to elicit the rules and procedures for approving a loan can there ever be no bias Why this might be difficult All views of reality are filtered All decision making is based partly on personal values Implicit knowledge There is no document in which the rules for approving loans are written down Types Conflicting information Different bank staff have different ideas about what the rules are of bias Motivational bias Say do problem expert makes accommodations to please the interviewer or some other audience The loan approval process described to you by the loan approval officers is quite different from your observations of what they actually do Observational bias Probe effect Limitations on our ability to accurately observe the world The loan approval process used by the officers while you are observing is different from the one they normally use Cognitive bias Bias Mistakes in use of statistics estimation memory etc The loan approval officers fear that your job is to computerize their jobs out of existence so they are deliberately emphasizing the need for case by case discretion to convince you it has to be done by a human Easterbrook 2004 is bias Bias only exists in relation to some reference point Notational bias Terms used to describe a problem may affect our understanding of it 4 Easterbrook 2004 3 Department of Computer Science Bias Examples of Bias Social pressure response to verbal and non verbal cues from interviewer Group think response to reactions of other experts Impression management response to imagined reactions of managers clients Wishful thinking response to hopes or possible gains Appropriation Selective interpretation to support current beliefs Misrepresentation expert cannot accurately fit a response into the requested response mode Anchoring contradictory data ignored once initial solution is available Inconsistency assumptions made earlier are forgotten Availability some data are easier to recall than others Underestimation of uncertainty tendency to underestimate by a factor of 2 or 3 5 1 University of Toronto University of Toronto Department of Computer Science Elicitation Techniques Traditional techniques Discourse Analysis Interviews Conversation Analysis Speech Act Analysis Open ended Structured Soft Systems Analysis Meetings Helps to prepare for other types of fact finding Cognitive techniques Focus Groups Brainstorming JAD RAD workshops Prototyping Participatory Design Advantages Helps the analyst to get an understanding of the organization before meeting the people who work there Sociotechnical Methods Surveys Questionnaires Sources of information company reports organization charts policy manuals job descriptions reports documentation of existing systems etc Participant Observation Enthnomethodology Analyzing hard data Collaborative techniques Ethnographic techniques Reading existing documents Background Reading Contextual social approaches Introspection Department of Computer Science e g by being aware of the business objectives of the organization Task analysis may provide detailed requirements for the current system Protocol analysis Knowledge Acquisition Techniques Card Sorting Laddering Repertory Grids Proximity Scaling Techniques Disadvantages written documents often do not match up to reality Can be long winded with much irrelevant detail Appropriate for Whenever you not familiar with the organization being investigated 6 Easterbrook 2004 University of Toronto Department of Computer Science Hard Data and Sampling Hard data includes facts and figures University of Toronto Example of hard data 7 Department of Computer Science Questions Forms Invoices financial information Reports used for decision making Survey results marketing data Easterbrook 2004 What does this data tell you Sampling What would you do with this data Sampling used to select representative set from a population Purposive Sampling choose the parts you think are relevant without worrying about statistical issues Simple Random Sampling choose every kth element Stratified Random Sampling identify strata and sample each Clustered Random Sampling choose a representative subpopulation and sample it Sample Size is important balance between cost of data collection analysis and required significance Process Easterbrook 2004 Decide what data should be collected e g banking transactions Determine the population e g all transactions at 5 branches over one week Choose type of sample e g simple random sampling Choose sample size e g every 20th transaction 8 Easterbrook 2004 9 2 University of Toronto University of Toronto Department of Computer Science Interviews Interviewing Tips Types Structured agenda of fairly open questions Open ended no pre set agenda e g the weather the score in last night s hockey game e g comment on an object on the person s desk My what a beautiful photograph Did you take that Advantages e g How long have you worked in your present position Follow up interesting leads E g if you hear something that indicates your plan of action may be wrong Unanswerable questions how do you tie your shoelaces Tacit knowledge and post hoc rationalization Removal from
View Full Document
Unlocking...