Design-Based Implementation Research and Big Data (#bigdata)
Educational data mining and rapid A/B testing can produce information for refining and enhancing digital learning systems, but they are less than ideal for answering questions about how digital learning systems are being used in different contexts and how implementation variations relate to differences in outcomes. An emerging research approach that is suited for this kind of inquiry is design-based implementation research (DBIR).
Design-Based Implementation Research
DBIR is an approach for investigating learning outcomes and implementation in tandem. It seeks to change the relationship between research and practice so that interventions are designed from the start with their ultimate uses in mind and are based on theories and methods from both the learning sciences and policy research. Penuel et al. (2011) articulated four core DBIR principles:
- focus on a persistent problem of practice;
- commitment to iterative,
- collaborative design;
- concern with developing theory and knowledge concerning both classroom learning and implementation processes; and concern with developing capacity for creating sustainable education system change.
With its roots in several decades of design research (Kelly, Lesh, and Baek 2008), DBIR calls for sustained partnerships between developers, education researchers, and practitioners who jointly select a problem to work on and engage in multiple cycles of design and implementation decisions with data collection and analysis embedded in each cycle so that implementation can be refined based on evidence (Penuel et al. 2011).
DBIR is a complement to such techniques as educational data mining and A/B testing. One of its strengths—and a feature that the other two approaches lack—is the collection of information on what learners and their teachers, peers, and others in their environments are seeking to accomplish and what they are doing before, after, and during learning sessions. When the learning session includes digital interaction, the digital learning system can collect data automatically, and those data can be combined with the knowledge collected by practitioners or researchers in the offline world for a more complete picture.
An example of the complementarity of contextual and learning system data comes from the work of Carnegie Learning, a publisher of math curricula for middle school, high school, and postsecondary students. A school was using its tutoring system as part of a mandated school improvement effort. Examining data collected automatically by the tutoring system, Carnegie Learning analysts could see that students in most classes were progressing as expected but that students in one class had stopped making gains midyear. When they brought this pattern of data to the attention of the school principal, they learned that the class that had stalled had lost its regular teacher and was being handled by a substitute. Seeing the data from the tutoring system, the principal realized that students in this class were suffering and decided that the plan to delay hiring a replacement teacher had to be changed as quickly as possible (Ritter 2012).
A relatively mature example of DBIR principles is the work of the Pathways project being led by the Carnegie Foundation for the Advancement of Teaching. This example illustrates the importance of implementation research in improving not just the design of a course with a strong technology component, but also the institutional practices in its implementation. The primary goal of this work is to improve outcomes for developmental mathematics students in community colleges in terms of entry into and success in college-level mathematics courses—a broader, more consequential objective than demonstrating that the online course per se produces mathematics learning.
Tony Bryk, President of the Carnegie Foundation for the Advancement of Teaching, launched the Pathways project in part to create a concrete example of how education research can lead to educational improvement. Similar to technology developers from industry, who embrace continuous improvement, Bryk views rapid cycles of modification, analysis of results, and redesign as key to improvement. Bryk argues that improvement research should be structured as many rapid iterations of small changes, what he calls “rapid iterative small tests of change.” His reasoning is that small changes can be implemented quickly, can be tested repeatedly in multiple contexts to make sure they are really improvements, and are unlikely to do harm (thus managing the risk associated with failure). By implementing many iterations in a short time, research collaborations can produce dramatic change through the accumulation of many small improvements.
Bryk further argues that traditional large-scale education research is most useful in few circumstances. He characterizes the research space in terms of three dimensions: confidence that a proposed change will lead to improvement (high or low); risk, or cost of failure (large or small); and the current situation with respect to stakeholders’ receptivity to the change (resistant, indifferent, ready). Of the 12 possible combinations of these dimensions, in Bryk’s view only two combinations (high confidence, indifferent audience, small cost; and high confidence, ready audience, and large cost) warrant a large-scale formal study (Bryk 2011).
Uses of Evidence from Implementation Research
DBIR proponents work with their practitioner partners to lay out a theory of the implementation steps needed in the practitioners’ context and study the implementation processes and outcomes simultaneously. The evidence they typically seek is correlational patterns, and they use quasi- experimental designs rather than RCTs, though some DBIR studies include experimental tests of different strategies for supporting implementation.
The Pathways project (described in the example Collaborative Research and Development on the Pathway to College Math) has emphasized investigation of the relationships between specific changes in practices and changes in student completion rates for the developmental math sequence. The lack of alternative plausible explanations for dramatic changes in an outcome (in the Pathways project, dramatic differences from historical rates in the numbers of students qualifying for college-level mathematics by their second year of college) gives some credence to causal inferences, even in the absence of a random-assignment experiment. In some examples of DBIR, alternative plausible explanations exist for observed differences, and important decisions hang in the balance, making it appropriate to incorporate experimental studies into DBIR.
Example: Collaborative Research and Development on the Pathway to College Math
When researcher Tony Bryk became President of the Carnegie Foundation for the Advancement of Teaching, he wanted to increase the impact of education research (Bryk, Gomez, and Grunow 2011). He and his colleagues argued that research should focus on a “persistent problem of practice” that, if solved, could have significant benefits for the education system. Bryk and colleagues often refer to these as high-leverage problems.
One such persistent problem is the developmental mathematics courses that students must take if they enter college not yet ready for college-level math. Many students believe developmental math courses just repeat their high school math experience. At some colleges, the lowest scoring students are required to pass as many as three semesters of developmental math courses (for which they do not earn credit) before being allowed to take credit-bearing college math courses. Not surprisingly, as many as 70 percent of these students become discouraged and fail to complete all the required developmental math courses. Without completing these requirements, they cannot earn a degree.
The Carnegie team defined its goal as doubling the number of students who earn college math credit within one year of continuous enrollment. To achieve this goal, the Carnegie team set out to collaborate with college administrators and instructors to redesign their approach to developmental mathematics by developing new courses and associated polices and then improving the new courses and practices by analyzing system data and feedback from implementation. They recognized that such an effort would need a collaborative community and an infrastructure to support its success.
Community and four-year colleges were invited to participate in a networked improvement community (NIC) for developmental math. A NIC is a group of people from multiple organizations committed to working together on a complex, high-leverage problem with a concrete target and a shared set of inquiry practices, including using what they build. The colleges that Carnegie convened agreed to collaborate with other colleges and with researchers and developers to implement the resulting new developmental math curriculum with their students, share data from their implementation, and participate in discussing implementation data and planning refinements. NIC participants recognized that as their work unfolded, new aspects of problems would become visible, and the NIC colleges found themselves working on emergent issues such as student engagement and persistence and the elimination of language that is a barrier to mathematics learning.
One of the important NIC activities was an analysis of the causes of the high failure rate for developmental math at their institutions. The collaborators found that many students were lost at the transition between multiple courses in a series, that the developmental math courses were not engaging, that many students had negative beliefs and attitudes about their ability to do math, and that many students’ ties to peers, faculty, and programs of study were weak. Among the strategies that the group decided to apply to address these issues was consolidation of what had been multiple math courses into a single course emphasizing real-world problems from statistics. The Pathways project has worked on two courses: Statway, which deals with developmental math content in the context of statistics, and, more recently, Quantway, a course on quantitative reasoning and literacy.
The Statway development process illustrates how educators, developers, and researchers can collaborate to iteratively co-design a new intervention. A small group of academic researchers and curriculum developers produced the initial version of Statway. Community college faculty reviewed this initial version and informally tried out some of the lessons from it with their students in fall 2010. Ongoing conversations among researchers, course designers, and math faculty led to the conclusion that this first version needed a major reworking. A team of college math faculty members was brought to Carnegie to redesign the course, and the result was Statway Version 1.5, which was pilot-tested NIC-wide in school year 2011–12.
Statway uses the OLI course engine to support its homework platform. This course engine made it possible to obtain detailed learning data on students’ engagement with individual problems and their persistence through the problem sets. Louis M. Gomez, a Learning Sciences professor at UCLA and Statway collaborator, expects that these data will enable the NIC to explore how various practices (implementation and context variables) make a difference in Statway outcomes and whether they vary by local setting.
When asked whether the Pathways project had conducted an efficacy study comparing Statway results with those for conventional developmental math sequences, Gomez explained,
We haven’t done an experiment on Statway versus business as usual at a community college. Right now our goal is to improve Statway and have it be executed reliably in the variety of contexts that make up the NIC. We need to do more than convince ourselves that it works. All kinds of promising interventions are subjected to RCTs that show nothing; often because they’re subjected to [experimental studies] too early. Equally important to work on is getting your intervention to work reliably across many different contexts. This is more important at this point than understanding whether Statway works better or worse than some other approach.
Gomez pointed out that he does not view comparative experimental research as “wrong” but useful for answering a different kind of question. Having defined its task as improving rates of successful completion of developmental mathematics, the Carnegie team is more focused on understanding how to get Statway to produce this outcome in a range of college contexts (external validity) than on comparing it with alternative approaches in an experimental design (internal validity).
Gomez’s colleague Paul LeMahieu noted that in the first year of Statway implementation, three times as many students earned a college math credit in one-third the time compared with historical averages at the participating colleges.
U.S. Department of Education, Office of Educational Technology, Expanding Evidence Approaches for Learning in a Digital World, Washington, D.C., 2013.