How to Leverage xAPI for Business Goals
The birth of the latest learning standard Experience API (xAPI) is exciting. It’s been boasted that the sky is the limit for what xAPI can do to liberate learning designs outside the restraints of any learning system. The truth, however, is that xAPI isn’t a magic wand. More precisely, xAPI is a tool for learning designers or performance “detectives” to answer questions such as “Does this plan (with activities across disparate applications) work well to meet the training target? Where can it be improved? Who isn’t doing well with this plan and why?”
An xAPI project should start with asking questions; questions reveal problems, needs, insights, assumptions. Then questions critical to prioritized business metrics should be identified and a deployment be designed to dig out how to improve those metrics. It’s also important to consider the availability of xAPI data. It’s suggested one start with well-defined questions that have business values and for which necessary data are available in short term; continuously iterate toward metrics targets (some questions answered and new questions emerged); and then expand the project scope.
Almost all xAPI statements are triggered by UI events, from a learner playing a video to an instructor evaluating a submitted work. A simple UI event can generate dozens of lines of code. Learning Record Store is only a big data storage (ADL has released open source code); without visualization as a cognitive agent, users can do nothing with xAPI data. How can we make the process efficient for most relevant roles? Most learning designers or project managers aren’t xAPI/dataviz experts; they should not worry about designing xAPI statements and visualizing them. That’s our belief when we built Visca – xAPI visual analytics service.
Big Data Approach Replaces Traditional Research Method
In today’s digital learning, there is a huge opportunity to leverage big data much more efficiently than ever. Quoted from the paper by Bill Cope & Mary Kalantzis “Interpreting Evidence-of-Learning: Educational research in the era of big data“:
The ‘big data’ approach is to collect data through practice-integrated research. If a record is kept of everything that happens, then it is possible analyze what happened, ex post facto. Data collection is embedded. It is on-the-fly and ever present. With the relevant analysis and presentation software, the data is readable in the form of data reports, analytics dashboards and visualizations.
This paper focuses mainly on what the authors call “semantically legible” structured data where data points have been designed into the learning environment. “Semantically legible” is self-describing, structured data, and is “learner-actionable feedback”. Every such data point can offer an opportunity that presents to the learner a “teachable moment“. These data points can involve either or both a machine response to learner action or machine-mediated human response, thereby harnessing both collective human intelligence and artificial intelligence.
Current software development methodology termed ‘agile development,’ which emphasizes rapid, iterative, frequent and incremental, design, testing and release cycles, has been applied to scholar research.
In our Scholar project, for instance, the research and development cycles last two weeks. ‘User stories’ are generated based on issues arising in the previous two weeks of implementation, coding occurs, then after two weeks they are either released to n = all, or to a subgroup if we want to compare A/B effects to decide whether the change works for users. Research and design are fluid, re-planned every two weeks. Research happens in ‘real time’, rather than fixed, pre-ordained researcher time.
In the era of big data, there is no need to figure or justify an optimal sample size, because there is no marginal cost of making n = all. There are no possibilities of sample error or sample bias. With n = all and self-explanatory dataviz available in real time, we do not need to support our methodology with power analyses. In other words, we relay much less on inferential statistics modeling (it’s not real time, with sample bias) and more on direct reasoning because users can see each data point needed to answer questions about individuals or groups.
All these arguments resonate with what we are doing with xAPI and Visca. Structuring data statements, presenting them, and putting data into use in real time (i.e. action taking or automation of flow) are our first priorities. Of course, the data can be used later on for different purposes at different scales or time frames, including training artificial intelligence to assist humans.
Standardized Visualization for Efficient Team Communication
Self-explanatory dataviz available in real time is also crucial for efficient communication and collaborative decision-making for cross-department and interdisciplinary team work, which is how most xAPI projects work because real-world problem-solving is always complex. Data-driven decision making involves ongoing build-measure-learn cycles, it’s a knowledge-building process in an organization.
The expert isn’t one person, it’s the community of minds (cited from the concept by John Seely Brown). Standardized visualization helps the community talk on the same page and connect thoughts easily; it’s not a brain but a network of brains that will solve organization challenges. That’s why we think standardized visualization for basic modules is necessary.
Start from the Most Influential Factors for Behavior Engineering
Some xAPI consultants might tell you xAPI enables us to evaluate training effectiveness at all five levels. A modified Kirkpatrick model is drawn for you. But wait, that still confines your viewpoint and thinking in a training setting.
The Behavior Engineering Model (BEM) developed by Gilbert and presented in his landmark book, Human Competence: Engineering Worthy Performance (Gilbert, 1978, p. 88), provides us with a way to systematically and systemically identify barriers to individual and organizational performance. The BEM distinguishes between a person’s repertory of behavior (what the individual brings to the performance equation) and the environmental supports (the work environment factors that encourage or impede performance). Roger Chevalier had adapted the BEM to provide a more efficient method for troubleshooting performance and for discovering the most important opportunities for improving individual performance. (link to paper)
Examining the potential impact that a change would make and the cost associated with that change, the leverage effect is illustrated in Fig. 3. Environmental factors should be the starting point for analysis because they pose the greatest barriers to exemplary performance. Training, which only builds knowledge, has the least leverage effect.
We suggest structuring and prioritizing xAPI projects after evaluating different factors that impact individual performance and business goals with the help of the BEM framework. xAPI can pull data from all systems (i.e. CMS, CRM, help desk, HRIS, marketing tools, mobile applications, ERP, ESN… etc.) and integrate them for analysis. What really makes xAPI shine is that it enables real-time data integration from disparate systems, with pre-designed dataviz we can make real-time integrated data feedback/communication loops possible. Even though the necessary data isn’t in xAPI format, we need to get them and put them together. Visca allows you to import your own data to do correlation analysis (e.g. business metrics vs. employee’s behaviors), or you can export xAPI data from Visca into your database to analyze.
Actionable Metrics and Vanity Metrics
xAPI is a shiny Swiss knife for performance “detectives” to find answers to critical questions. But it can’t ask questions for you, it can’t do learning design for you, and it can’t do design thinking for problem solving, of course. A tool is only as effective as how you use it; don’t cut things randomly just because you have a knife. And don’t just think inside the viewpoint of training setting. You can’t manage what you don’t measure. What’s worth measuring and analyzing is everything that’s critical for business metrics, and identified relevant performance and behavior metrics, nothing more. Design the xAPI deployment to dig out the answers and right actions step by step. Design thinking is key.
Last, be aware of the difference between actionable metrics and vanity metrics. If a metric isn’t actionable and you can’t do anything to make it better because you can’t understand why it’s so, then it’s a vanity metric. Actionable metrics are data that tie to specific and repeatable tasks you can improve and to the goals of your business. Make sure you have the right metrics.
Related reading: xAPI and Gamification (a mini interactive web-based course)