The Power of Big Ideas in Mathematics Education: Development and Pilot Testing of POWERSOURCE Assessments CSE Report 697 David Niemi, Julia Vallone, and Terry Vendlinski CRESST/UCLA August 2006 National Center for Research on Evaluation, Standards, and Student Testing (CRESST) Center for the Study of Evaluation (CSE), Graduate School of Education & Information Studies University of California, Los Angeles GSE&IS Building, Box 951522 Los Angeles, CA 90095-1522 (310) 206-1532Copyright © 2006 The Regents of the University of California The work reported herein was supported under the Educational Research and Development Centers Program, PR/Award Number R305B960002, as administered by the Office of Educational Research and Improvement, U.S. Department of Education. The findings and opinions expressed in this report do not reflect the positions or policies of the National Institute on Student Achievement, Curriculum, and Assessment, the Office of Educational Research and Improvement, or the U.S. Department of Education.1 THE POWER OF BIG IDEAS IN MATHEMATICS EDUCATION: DEVELOPMENT AND PILOT TESTING OF POWERSOURCE ASSESSMENTS David Niemi, Julia Vallone, and Terry Vendlinski Center for Research on Evaluation, Standards, and Student Testing University of California, Los Angeles Abstract The characteristics of expert knowledge—interconnectedness, understanding, and ability to transfer—are inextricably linked, a point that is critically important for educators and constitutes a major theme of this paper. In this paper we explore how an analysis of the architecture of expert knowledge can inform the development of assessments to help teachers move students toward greater expertise in mathematics, and we present examples of such assessments. We also review student responses and preliminary results from pilot tests of assessments administered in sixth-grade classes in a large urban school district. Our preliminary analyses suggest that an assessment strategy based on the structure of mathematical knowledge can reveal deficiencies in student understanding of and ability to apply fundamental concepts of pre-algebra, and has the potential to help teachers remediate those deficiencies. Introduction For educators, researchers and others who are interested in improving learning, and particularly learning of complex skills and knowledge, a fundamental challenge is understanding how isolated skills and pieces of knowledge learned in a variety of classroom contexts can become inter-connected, meaningful, and generalizable. As cognitive science research has convincingly demonstrated, it is the connections among elements of knowledge, understanding of important concepts and principles, and the ability to apply knowledge flexibly and effectively in a wide variety of situations, that characterize the development of knowledge toward greater expertise (e.g., Ausubel, 1968; Bereiter & Scardamalia, 1986; Chi & Ceci, 1987; Chi, Glaser, & Rees, 1982; Glaser & Chi, 1988; Larkin, McDermott, Simon, & Simon, 1980; Niemi, 1996; NRC, 2002, 2004;2 Silver, 1981). This is true for all domains that cognitive scientists have studied, including mathematics, science, history, reading, writing, and other school subjects. The Importance of Big Ideas Decades of cognitive research and educational experience have shown that when specific responses to specific tasks or questions are learned by rote, that knowledge does not generalize (e.g., Bassok & Holyoak 1989a, b; Bransford, Brown, & Cocking, 1999; Carpenter & Franke, 2001; Chi, Glaser, & Farr, 1988; Ericsson, 2002; Larkin, 1983; Newell, 1990; NRC, 2004). Despite the robustness of this finding, however, mathematics instruction in the U. S. has historically focused on the memorization of specific responses to specific questions, and as a result, the knowledge most students have is extremely context bound and not generalizable. Most K-12 mathematics students do not construct the meaning of core concepts and principles, cannot relate concepts to problem-solving skills and procedures, and view mathematics as a collection of isolated, meaningless procedures to be memorized, not understood (e.g., Carpenter & Lehrer, 1999; Heibert & Carpenter, 1997; Porter, 1989; Schmidt, 2001; Stodolsky, 1988). In contrast to the piecemeal, context-bound knowledge that beginning learners have, expert knowledge has a relational structure: this is one of the strongest and most powerful conclusions to be drawn from decades of cognitive science research on the nature and development of knowledge—strongest, in the sense that it has been extensively and compellingly validated in a large number of studies, and powerful, in the sense that it has great explanatory force and broad implications for teaching, learning, and educational practice in general. For someone who has advanced knowledge in a domain, every element of that knowledge is connected to other elements in a highly organized structure, with certain statements, expressing important ideas, dominating and organizing other types of knowledge (e.g., Bereiter & Scardamalia, 1986; Chi & Ceci, 1987; Chi, Glaser, & Rees, 1982; Glaser & Chi, 1988; Larkin, McDermott, Simon, & Simon, 1980; Bransford, Brown & Cocking, 1999; Niemi, 1996; Wineburg, 2002). That certain ideas organize other kinds of knowledge, including problem-solving strategies and skills, was first and most dramatically revealed in a series of studies by Glaser and colleagues (Chi & Glaser, 1981; Chi et al., 1982). In one study, for example, when physics experts and novices were asked to sort problems printed on index cards (Chi et al., 1981), the experts put together problems on the basis of abstract concepts and principles, e.g., Newton’s laws, conservation of3 energy. Novices, on the other hand, sorted on the basis of physical features of the problem situation, e.g., “there’s an inclined plane in these problems”. The novices either did not understand the theoretical principles or did not know how and when to apply them to problem-solving situations. One effect of representing problems in terms of theoretical concepts is that expert problem solvers can activate and implement problem-solving procedures linked to those concepts, e.g., formulas for solving conservation of energy problems. Novices have to resort to remembering how they solved problems with similar surface features, which can lead to ineffective solution strategies as problems with the same