Learning by Teaching SimStudent: Technical Accomplishments and an Initial Use with Students∗ Noboru Matsuda1, Victoria Keiser1, Rohan Raizada1, Arthur Tu1, Gabriel Stylianides2, William W. Cohen1, Kenneth R. Koedinger1 1 School of Computer Science, Carnegie Mellon University. 5000 Forbes Ave. Pittsburgh PA 15213 USA 2 School of Education, University of Pittsburgh. 5517 Posvar Hall, Pittsburgh PA 15260 USA [noboru.matsuda, keiser, atu, wcohen, koedinger]@cs.cmu.edu [email protected], [email protected] Abstract. The purpose of the current study is to test whether we could create a system where students can learn by teaching a live machine-learning agent, called SimStudent. SimStudent is a computer agent that interactively learns cognitive skills through its own tutored-problem solving experience. We have developed a game-like learning environment where students learn algebra equa-tions by tutoring SimStudent. While Simulated Students, Teachable Agents and Learning Companion systems have been created, our study is unique that it genuinely learns skills from student input. This paper describes the overview of the learning environment and some results from an evaluation study. The study showed that after tutoring SimStudent, the students improved their performance on equation solving. The number of correct answers on the error detection items was also significantly improved. On average students spent 70.0 minutes on tu-toring SimStudent and used an average of 15 problems for tutoring. Keywords: SimStudent, Learning by teaching, tutor-learning effect, algebra equation solving, machine learning 1 Introduction There is ample evidence that students learn when they teach their peers [1]. Such an effect of tutor learning has been observed across different subjects, age groups, format of tutoring, and so forth. Yet, little is known about when tutors’ learning would be facilitated and why. A scientific contribution of the current study is at our exploratory effort to study cognitive and social factors for tutor learning. Even when tutor learn-ing is effective, there are practical difficulties to exercise peer tutoring in an actual classroom – not only would it be time consuming (the students must take turns) but, ∗ This study is supported by National Science Foundation Award No. DRL-0910176 and by Department of Education (IES) Award No. R305A090519. This work is also supported in part by the Pittsburgh Science of Learning Center, which is funded by the National Science Foundation Award No. SBE-0836012.also, the tutees might not learn as much as tutors do. Thus, on the engineering side of our contribution, building an effective and efficient learning environment that facili-tates tutor learning is one of our primary research goals. We have developed a game-like learning environment where students learn by in-teractively tutoring a computer agent, called SimStudent. SimStudent is a machine-learning agent that learns cognitive skills from examples and through its own tutored problem-solving experiences [2]. Our long-term research goal is to investigate the effect of tutor learning with SimStudent as a teachable agent. The aim of this paper is to provide an overview of our learning by teaching system and discuss results from an evaluation study. The primary research question in the current paper addressed whether or not the students learn by teaching SimStudent at all, and if so, how effective the system is. 2 Learning by Teaching 2.1 Type and Domain of the Proposed Learning by Teaching Environment The effect of tutor learning has been studied in many different domains, across ages, and in various tutoring settings [3]. Various forms of tutoring have been observed including reciprocal teaching [4] and collaborative passage learning [5]. The effect of tutor learning has been observed for all age groups including college [6], high school [7], middle school [8], and elementary school students [9]. The tutor learning effect has been shown to be relatively more effective in math than reading [3, 10]. In the current study, we focus on one-on-one tutoring where a single student acts as a tutor and a computer agent plays the tutee’s role. Although the SimStudent technol-ogy and the overall framework of the proposed learning environment are domain independent, the current learning system is built for algebraic linear equations – one of the more challenging subjects in mathematics. 2.2 Related Studies There have been a number of simulated students (also called teachable agents) devel-oped so far [11-14]. VanLehn et al. [11] developed one of the earliest simulated stu-dents and demonstrated its benefit for teacher training in physics. Betty’s Brain [15] and its variations are the most recent examples of a teachable agent used to study the tutor learning effect. Betty’s Brain learns causal relations from a conceptual map created by student by entering nodes (each representing a concept) and links (each representing a causal relation among the concepts). Students can also quiz Betty’s Brain with a problem asking a causal relation (e.g., “If dead organisms increase, what happens to the animals?”). While Simulated Students, Teachable Agents and Learning Companion systems have been created, some were never used with real students and others do not genu-inely learn from student input. While the VanLehn's system [11] incorporated ma-chine learning and could be used for theory generation and to analyze instructional materials, it was not designed for use with students. On the other hand, while Betty'sBrain has been used extensively by students and subject to numerous evaluations, it does not have a machine-learning component. Students teach Betty's Brain by editing a concept map, but the system does not learn from the concept map any more than making straightforward inferences from following the links in the map. This paper provides perhaps the first demonstration of a machine learning system being used as a teachable agent by real students and with significant pre-to-post learning outcomes. 3 SimStudent as a Teachable Peer Learner 3.1 Overview of SimStudent The underlying technology for SimStudent’s learning is inductive logic programming in the form of programming by demonstration [16,