Game Design Handbook from GlassLab
Meaningful games create powerful impacts on players, fostering a positive learning culture that enables expression and the ability to learn from failure. GlassLab’s Game Design Handbook serves as a living, quick, and actionable resource for learning game design for developers (though every person is a game designer at heart!). A good resource!!
We especially love this part, it’s true for designing any type of assessment no matter it’s game-based or not. But in a simulation-like and scenario-based experience that a learning game offers is more promising to be able to assess higher level thinking abilities and multiple intelligences.
When developing an assessment model for a game, a leap of logic is required when we try to move from the gathering of behavioral metrics to composing that behavioral data into meaningful and reliable indicators of a player’s performance or capability. Behavioral data is just noise, but reliable indicators — the pattern found in the data — make music. In the language of assessment, this is the difference between data and evidence.
Sprinkled throughout GlassLab’s work are references to ECD, or Evidence-Centered Design. This describes a deliberate effort to produce activities in the game which can produce evidence that lets you make inferences about a player’s knowledge and skills. Since we can’t peer into everyone’s head and see what they know, we have to infer it from evidence.
Data is NOT Evidence
From a data perspective, one major product of gameplay is a time-stamped log of interactions. This is the data, a raw record of learners’ interactions with the game. The big challenge then becomes determining how to take this raw record and extract information and apply rules that will tell us about a person’s knowledge or skill in a particular area.
This may be clearer if we look at how our traditional multiple choice tests do this. The data out of a multiple choice test is the string of the options that a student selected (say: a,b,a,c, etc.). We then apply a simple scoring rule (if the option selected equals the correct answer in the key, then correct, else incorrect) to indicate correctness, which is the evidence.
In the same way, game evidence might be whether something is correct or not, but it might also be a sequence of actions, a count of one or more actions, or the time spent on one or more actions. Evidence is what differentiates the play of a novice in the area of interest from someone who is proficient in your area of interest.
How to Know if You’re Producing Evidence
Evidence is not complicated. If your assessment plans for a given activity cannot be explained in just a couple minutes, you are probably going to fail to gather strong, actionable evidence. How do you know if you’re on the right track? Do a couple checks early on in development. Have a handful of players with different levels of skill in your area of interest play the game. Do they do different things depending on their proficiency? There is your evidence.
Evidence is Good Game Design
Player behaviors that are competency-related choices will tend to produce good evidence, which is rarely difficult to explain or understand. From an evidence perspective, these observations are most powerful when the choice is such that students who haven’t mastered a concept are likely to make one choice and students who have mastered it are likely to make the other. The more we can create a choice that differentiates between the groups, the stronger your evidence will be.
In “When Data Empowers the Player“, questions are asked: Is there a way of providing kids with a sort of ‘pivot table’ allowing them to see their progress over time, that’s deeper and more nuanced than just completion of levels, or collection of items? What if a game had a technology tree that reflected not just increase in strength and shield, but exciting and empowering statistics about children’s performance in math or literacy?
These great questions resonate with one of the important survey results from A-Games Project (How Learning Games Are Used to Support Formative Assessment) — teachers need competency standard oriented reporting instead of just points and levels in the game. If the assessments embedded inside the game are well tagged with metadata (competency standard, Bloom’s levels etc.), it’s possible to give teachers and students this kind of precise feedback and actionable reporting.