Three Things Game Designers Need to Know About Assessment (#GBL)
Princeton, N.J. (November 7, 2013) — Members of Educational Testing Service’s (ETS) Research & Development (R&D) division led the assessment research behind GlassLab’s new educational game SimCity™EDU: Pollution Challenge! This new game brings elements of computer gaming to the world of education — providing assessment and learning in an environment that is engaging and exciting. ETS researchers are collaborating with GlassLab on several more educational games.
“Today’s educational landscape is going through enormous changes. Our participation in GlassLab is one way to make sure that assessments built into new educational games rest on a solid research foundation, and support quality and equity in education and fair and valid assessments,” says Senior Vice President Ida Lawrence, who leads ETS’s R&D division.
Pollution Challenge! invites students to learn by resolving conflicts between economic development and environmental pollution in a virtual city. Students learn by reading, viewing and analyzing information in a formative assessment game that provides teachers and students with feedback that supports learning.
“Game-based assessment may look very different from traditional assessments, but they have fundamental principles in common. Games designed to work as assessments need to be based on psychometric principles,” says Robert Mislevy, holder of the Frederic Lord Research Chair at ETS and a member of ETS’s team of researchers working with GlassLab. “Assessments aim to gather and make sense of information about what students know and can do, and this information will be better the more engaged the students are, which means that games have the potential to create conditions that foster learning.”
Researchers at ETS’s R&D division led the assessment work at GlassLab, a project of the nonprofit Institute of Play, supported by the Bill & Melinda Gates Foundation and the John D. and Catherine T. MacArthur Foundation. Other collaborators in the GlassLab project include Electronic Arts™, the Entertainment Software Association®, and Pearson’s Center for Digital Data, Analytics and Adaptive Learning.
ETS is a world leader with long experience in creating fair and valid educational assessments for a variety of platforms. The collaboration with GlassLab allows ETS’s researchers to bring their expertise in assessment development, cognitive and learning science, validity research, and psychometrics to bear on a number of new and challenging issues surrounding game-based assessment. ETS researchers are involved in developing new assessment design and psychometric approaches and tools as part of the GlassLab collaboration.
“Game designers already think about what players have to learn as part of playing a game, but learning to play a game is not necessarily the same as learning from playing a game. GlassLab has re-conceptualized aspects of SimCity to focus on learning that is valued academically,” says Malcolm Bauer, a senior research scientist in ETS’s R&D division, and part of the GlassLab team. “We are targeting aspects of the Next Generation Science Standards, as well as components of the Common Core State Standards for ELA and Math.”
“We are discovering what it takes to create valid and scalable game-based assessments by marrying the design paradigms for game development, learning, and assessment,” says Andreas Oranje, psychometric director in the K-12, NAEP, and Global unit of Statistical Analysis within the R&D division.
“We are integrating several design frameworks,” Oranje says, pointing to the game design framework developed by Electronic Arts; Evidence Centered assessment Design developed at ETS by Mislevy and colleagues; and the software design framework Agile to create evidence-centered game design. “We have now begun to develop a deeper understanding of how game telemetry and analytics interact with our psychometric models.”
ETS’s researchers will continue to collaborate with GlassLab on several more game-based assessments. Some of them will be educational versions of existing games like SimCityEDU, while others will be entirely new games.
Read more about assessment and game design in ETS Research Spotlight No. 8. This essay summarizes Three Things Game Designers Need to Know About Assessment, which appeared as a chapter in a book on game-based assessment. The lead author of the original chapter was Mislevy. Part of the introduction states that:
The three things game designers need to know are:
1. Assessment and game design are based on the same principles of learning.
2. Assessment design isn’t fundamentally about tasks and scores, but about the structure of reasoning — reasoning from what students say and do, to understand what they know and can do.
3. Designers of game-based assessments must consider key constraints of both assessment and game design right from the start.
The 488-page volume has three parts. Part one provides the background for game-based learning and assessment and discusses what game designers need to know in order to build games that also are valid and efficient tools of assessment. Part two explores new ways of evaluating learning in computer-based gaming environments that can provide both individualized assessment and feedback to learners. Part three explores the latest empirical research findings and successful examples of game-based assessment.
At ETS, we advance quality and equity in education for people worldwide by creating assessments based on rigorous research. ETS serves individuals, educational institutions and government agencies by providing customized solutions for teacher certification, English language learning, and elementary, secondary and post-secondary education, as well as conducting education research, analysis and policy studies. Founded as a nonprofit in 1947, ETS develops, administers and scores more than 50 million tests annually — including the TOEFL® andTOEIC ® tests, the GRE ® tests and The Praxis Series™ assessments — in more than 180 countries, at over 9,000 locations worldwide.www.ets.org
A project of Institute of Play, GlassLab brings together leaders in commercial games and experts in learning and assessment to leverage digital games as powerful, data-rich learning and formative assessment environments. The Lab represents a groundbreaking collaboration between Institute of Play, the Entertainment Software Association, Electronic Arts, Educational Testing Service, Pearson’s Center for Digital Data, Analytics & Adaptive Learning and others. GlassLab is supported by the Bill & Melinda Gates Foundation and the John D. and Catherine T. MacArthur Foundation. More information is available at http://glasslabgames.org.