Task Templates Based on Misconception Research CSE Report 646 Jennifer G. Cromley and Robert J. Mislevy National Center for Research on Evaluation, Standards, and Student Testing (CRESST)/ University of Maryland, College Park December 2004 Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing Graduate School of Education & Information Studies University of California, Los Angeles Los Angeles, CA 90095-1522 (310) 206-1532Project 3.6 Study Group on Cognitive Validity, Strand 1 Cognitively Based Models and Assessment Design Project Director: Robert J. Mislevy, CRESST/University of Maryland, College Park Copyright © 2004 The Regents of the University of California The work reported herein was supported under the Educational Research and Development Centers Program, PR/Award Number R305B960002, as administered by the Institute of Education Sciences, U.S. Department of Education. The findings and opinions expressed in this report do not reflect the positions or policies of the National Center for Education Research, the Institute of Education Sciences, or the U.S. Department of Education.1 TASK TEMPLATES BASED ON MISCONCEPTION RESEARCH1 Jennifer G. Cromley and Robert J. Mislevy CRESST/University of Maryland, College Park Abstract Researchers spend much time and effort developing measures, including measures of students’ conceptual knowledge. In an effort to make such assessments easier to design, the Principled Assessment Designs for Inquiry (PADI) project has developed a framework for designing tasks and to illustrate its use has “reverse engineered” several existing science assessments. This paper reports one such project, motivated by assessments that elicit students’ qualitative explanations of situations that have been designed to provoke misconceptions and partial understandings. We describe four task-specific templates we created—three based on Hestenes, Wells, and Swackhamer’s (1992) Force Concept Inventory and one based on Novick and Nussbaum’s (1981) Test About Particles in a Gas (TAP). We then describe an overarching framework for these templates, another PADI object called a Design Pattern, based on Stewart’s concept of “Model Using” (Stewart & Hafner, 1994). For each template, we describe a multivariate student model, a measurement model, and a task model. We conclude by suggesting how these templates and the design pattern could help researchers (and perhaps teachers) who wish to design new assessments in science domains where students are known to hold misconceptions. 1.0 Introduction Creating tasks to assess underlying concepts and inquiry processes in science is not an easy thing to do. The National Science Foundation has funded the Principled Assessment Designs for Inquiry (PADI) project, under the Interagency Educational Research Initiative (IERI), to create a conceptual framework and supporting software to help people design inquiry assessments. Among the data structures PADI has developed to this end are design patterns, which lay out assessment arguments at a conceptual level; task templates, which are schemas for the operational elements of an assessment, and support the creation of families of related tasks; and task specs, which describe the elements of individual tasks in transportable formats (specifically, the IMS/QTI standards and extensions thereof). One type of activity for the PADI project has been to take existing assessments used for science inquiry research, and to write templates and design specifications 1 Thanks to Rick Elliott for preparing the manuscript.2 for those assessments. We refer to this process as “reverse engineering,” in that the templates and design specifications that are developed starting from existing tasks could be used to reproduce the same assessments, or to produce new or analogous questions in the same or a similar domain. This report presents the results of applying reverse engineering to two conceptual assessments in science: the Force Concept Inventory (FCI; Hestenes, Well, & Swackhamer, 1992) and the Test About Particles in a Gas (TAP; Novick & Nussbaum, 1981). Both assessments are based on research about student misconceptions, an area of cognitive psychology research on expert and novice performance. Section 2 of the report provides a brief review of the theoretical basis for these assessments in the novice-expert and misconceptions paradigms in cognitive psychology. Section 3 then discusses the challenges that misconceptions research poses for assessment, and the benefits of making available to researchers the type of design patterns and templates that PADI is producing. Section 4 summarizes the structure of PADI task templates. Section 5 discusses the development of a design pattern and template for the Force Concept Inventory (FCI; Hestenes et al., 1992), a conceptual assessment of knowledge about Newtonian physics. The student model, evidence rules, statistical model, and task model for the template are discussed. Section 6 describes the process of adapting the template for the Test About Particles in a Gas (TAP; Novick & Nussbaum, 1981). Section 7 closes with a summary of the potential benefits to users of the design pattern and of the four templates that were developed. 2.0 Novice-Expert Research One of the dominant strains of cognitive psychology research from the mid-1970s through the 1980s was the study of expert performance across numerous domains (for reviews, see Charness & Schultetus, 1999; Ericsson & Charness, 1997). The basic premise of this line of research was that if the characteristics of expert performance could be isolated and identified, then perhaps novices could be trained in those specific knowledge, skills, and attitudes, in order to move them closer to expert performance. Moving away from previous notions of expertise as general and inborn, cognitive psychologists conducted research that led them to see expertise as domain specific and acquired through extensive teaching and practice (Ericsson, 1996; Ericsson & Smith, 1991). Though early expert-novice research tended to consider only expert performance (e.g., in chess, de Groot, 1946/1978), later research3 often contrasted experts and novices (e.g., in physics, Larkin, McDermott, Simon, & Simon, 1980). Early research also focused on problem solving in domains with clearly delimited solutions and limited solution paths, called