Psychometric Considerations in Game-Based Assessment
The Institute of Play has released a white paper called “Psychometric Considerations in Game-Based Assessment”, it is a work of multidisciplinary scholarship. It’s also the first publication of multidisciplinary collaboration. GlassLab brings together game designers, learning scientists, and psychometricians from Institute of Play, the Entertainment Software Association, Electronic Arts, Educational Testing Service, Pearson’s Center for Digital Data, Analytics & Adaptive Learning, and other organizations.
A primer, of sorts, for the emerging fi eld of game-based assessment, the paper focuses specifi cally on the formative assessment value of simulation games, and in particular, the game SimCityEDU: Pollution Challenge!. Using the game as a case study, the paper explores the ways in which psychometric considerations specifically, and assessment design more generally, can be integrated into the game development process.
This book is a companion to “Three things game designers need to know about assessment” (Mislevy, Behrens, DiCerbo, Frezzo, & West, 2012), which, by the way, are:
- The principles of assessment design are compatible with the principles of game design. It is because they both build on the same principles of learning.
- Assessment isn’t really about numbers; it’s about the structure of reasoning.
- The key constraints of assessment design and game design need to be addressed from the very beginning of the design process.
We discuss here what assessment designers and psychometricians need to know about game-based assessment—more broadly, how to think about a given game-based assessment (GBA) from the perspective of a psychometrician, but integrated with the key ideas and goals of other domains that are fundamentally important to its success. The first half of the piece doesn’t look like something out of Psychometrika or the Journal of Educational Measurement. We need to return to fundamentals of learning and design, of assessment arguments and evidentiary reasoning, to understand when and how the underlying concepts of psychometrics can be useful in GBA. We can then see how to integrate the concepts into GBA design from the beginning, and apply, adapt, or invent machinery to put them to work.
Chapters 1-5 are background that is meant to be broadly accessible, to game designers, subject-area experts, and learning scientists as well as measurement specialists. Chapter 6, on assessment design, is pivotal: It provides a conceptual framework for assessment design, which at once connects the game and learning aspects of a GBA with the assessment aspects, and gives meaning to psychometric modeling that may follow. Chapters 7-12 are more aimed at the measurement specialist, especially Chapter 10. We have tried to make these chapters readable, and we hope useful, to motivated readers from the allied fi elds. Chapter 13 discusses implications for GBA design, drawing on practices from the game industry, instructional science, and assessment design, and our own experience in trying to integrate them.
Related reading: From Digital Desert to Digital Ocean