USER, COME BACK!!Maxine EskenaziLanguage Technologies Institute, CMU June 16, 1999March 31, 20042CarnegieMellonWhat is the goal?Assessment differs when the application becomes real Then you can based it on the volume of callsAs long as we do not make real reservations, is user satisfaction valid?Users can point to places ( hotspots ) where they had a hard timeWhat do they remember that would make them call or not call again?We would want the system with the least hotspotsMarch 31, 20043CarnegieMellonSystem hotspotsWhat do you remember about the last system you called?It kept repeating the information about one hotel.Why was it irritating?I had to repeat myselfThe system asked the same thing over and overI told it Pittsburgh and then it asked where I wanted to goIt wasn t talking about the same thing I wasMarch 31, 20044CarnegieMellonMeasureslabel turns by subgoal (departure time, hotel location)sr=no. syst repeats; ur=no. user repeats; T=no. turns; us=user then syst said same thing; sn=syst subgoal /= user subgoalrepeats (R)= sr+ur/2T-2 (per goal)I had to repeat myself , The system asked the same thing overand over :shadows (S)= us/(T-1) (per goal)I told it Pittsburgh and then it asked where I wanted to gosystem/=user (N)= sn/TIt wasn t talking about the same thing I wasMarch 31, 20045CarnegieMellonSystem AssessmentCompare semi-automatic measures to what users sayPanel of 9 callers9 different scenarios, 3x3 different difficulty levelseasy = 1 leg; med. = 2 legs+; hard = 3 legs+each caller did: 1 easy, 1 medium, 1 hardMarch 31, 20046CarnegieMellonRepetitions - questionsTell me where in the dialogue:Something didn t go as you thought it shouldYou had to change what you wantedYou wanted to give upYou had no idea what to do nextMarch 31, 20047CarnegieMellonComparative resultsFor 20 dialogues, user/label agreement:highest number of turns: 65%highest percent of repeats (R): 85%highest percent of system shadows (S): 60%highest percent of system/=user (N): 55%(S) seems to refine the information that is in (R)divides problem repeats from simple navigationMarch 31, 20048CarnegieMellonComparative Results - Task DifficultyTurns Repetitions Shadows S/=UAll 65% 85% 60% 55%Easy 50% 100% 100% 50%Medium 62.5% 75% 62.5% 75%Hard 80% 40% 40% 20%March 31, 20049CarnegieMellonAssessment - Sanders questionnaire task successusefulnessefficiencyASRunderstandingTTScomprehensibleresponse times: understood all infouser initiativeknow what to say0 1 2 3 4 5March 31, 200410CarnegieMellonSanders questionnaire - cont.u: got all infohow to get infogot info efficientlyrealise there is a pbcan be successfully correctedeasy to correct0 1 2 3 4 5March 31, 200411CarnegieMellonLength of dialogues0 200 400 600 800MEANeasymediumhardshortestlongestMarch 31, 200412CarnegieMellonAcquired DataType No. Dialogs No. Utts. TimeHuman-Human 58 1800 ~1 hrWOZ1 107 1992 ~1.3 hrsWOZ2 16 487 ~20 minMovieline 112 7527 ~3 hrsSystem 2861 44783 ~11.4 hrsThis document was created with Win2PDF available at http://www.daneprairie.com.The unregistered version of Win2PDF is for evaluation or non-commercial use