In the Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005)Amsterdam, the Netherlands, 18-22 July, 2005An architecture to combine meta-cognitive andcognitive tutoring: Pilot testing the Help TutorVincent Aleven, Ido Roll, Bruce McLaren, Eun Jeong Ryu, Kenneth KoedingerHuman-Computer Interaction InstituteCarnegie Mellon University5000 Forbes Ave, Pittsburgh PA 15213, USAAbstract Given the important role that meta-cognitive processes play in learning,intelligent tutoring systems should not only provide domain-specific assistance, butshould also aim to help students in acquiring meta-cognitive skills. As a step toward thisgoal, we have constructed a Help Tutor, aimed at improving students’ help-seekingskill. The Help Tutor is based on a cognitive model of students’ desired help-seekingprocesses, as they work with a Cognitive Tutor (Aleven et al., 2004). To provide meta-cognitive tutoring in conjunction with cognitive tutoring, we designed an architecture inwhich the Help Tutor and a Cognitive Tutor function as independent agents, to facilitatere-use of the Help Tutor. Pilot tests with four students showed that students improvedtheir help-seeking behavior significantly while working with the Help Tutor. Theimprovement could not be attributed to their becoming more familiar with the domain-specific skills being taught by the tutor. Although students reported afterwards that theywelcomed feedback on their help-seeking behavior, they seemed less fond of it whenactually advised to act differently while working. We discuss our plans for anexperiment to evaluate the impact of the Help Tutor on students’ help-seeking behaviorand learning, including future learning, after their work with the Help Tutor.IntroductionA number of instructional programs with a strong focus on meta-cognition have been shown tobe effective, for example programs dealing with self-explanation (Bielaczyc, Pirolli, & Brown,1995), comprehension monitoring (Palincsar & Brown, 1984), evaluating problem-solvingprogress (Schoenfeld, 1987), and reflective assessment (White & Frederiksen, 1998). Theseprograms were not focused on the use of instructional software. Based on their success, onemight conjecture that intelligent tutoring systems would be more effective if they focusedmore on the teaching of meta-cognitive skills, in addition to helping students at the domainlevel. A number of efforts have focused on supporting meta-cognition in intelligent tutoringsystems (Aleven & Koedinger, 2002; Bunt, Conati, & Muldner, 2004; Conati & VanLehn,2000; Gama, 2004; Luckin & Hammerton, 2002; Mitrovic, 2003). In some of these projects,the added value of supporting meta-cognition was evaluated. Aleven and Koedinger showedthat having students explain their problem-solving steps led to better learning. Gama showedadvantages of having students self-assess their skill level. Still, it is fair to say that ITSresearchers are only beginning to evaluate the value of supporting meta-cognition in ITSs.Our research concerns help seeking. There is evidence that help seeking is an importantinfluence on learning (e.g., Karabenick, 1998), including some limited evidence pertaining tolearning with interactive learning environments (Aleven et al., 2003; Wood & Wood, 1999).We focus on the hypothesis that an ITS that provides feedback on students’ help-seekingbehavior not only helps students to learn better at the domain level but also helps them tobecome better help seekers and thus better future learners. We are not aware of anyexperiments reported in the literature that evaluated the effect that instruction on help-seekingskill has on students’ learning and their ability to become better help-seekers in the future.In order to test this hypothesis, we have developed a Help Tutor, a plug-in tutor agent(Rich et al., 2002; Ritter, 1997) that evaluates students’ help-seeking behavior and providesfeedback, in the context of their work with a Cognitive Tutor (Koedinger et al., 1997). Indeveloping such a tutor, there are a number of open issues. First, what exactly constitutes goodhelp-seeking behavior? At one level, it seems quite clear that students should workdeliberately, refrain from guessing, use the tutor’s help facilities when needed and only then(for example, when a step is unfamiliar or after repeated errors), and read problem instructionsand hints carefully. However, it is not always easy to know when help-seeking behavior isineffective and detrimental to learning. For example, Wood and Wood (1999) describe astudent who appeared to be requesting help from the system far too often, yet ended up withhigh learning gains. Furthermore, tutor development requires a detailed model that definesprecisely what it means, for example, to work deliberately or to use help only when needed.The creation of such a model is a research contribution in itself. We use the model that isdescribed in (Aleven et al., 2004). Since then it has been modified so that it captures a widerrange of students’ help-seeking strategies and provides feedback on only the most egregiousdeviations from reasonable help-seeking behavior.Second, how should the Help Tutor and the Cognitive Tutor be coordinated, especiallywhen both tutors might have conflicting “opinions” about the student’s action? An action canbe correct on the domain level but erroneous according to the Help Tutor and vice versa. Thereare many coordination options, with potentially significant effect on students’ learning, andvery few guidelines for selecting from them. In this respect, our work has similarities to thework of Del Soldato and du Boulay (1995) whose system, MORE, coordinated the advice of adomain planner and a motivational planner. The domain planner of MORE would typicallysuggest that a student tackle harder problems as they succeed on easier ones, while itsmotivational planner might suggest repeating easier problems to improve a student'sconfidence and level of success.Third, what kind of architecture can support combined cognitive and meta-cognitivetutoring? Our goal was to use the Help Tutor as a plug-in tutor agent that could be added to