The rising of MOOCs brings the interest of scrutinizing the effectiveness of online learning. Definitely it’s not a new-born baby, even MIT’s OpenCourseWare is more than 10 years old. But the Web2.0 technologies and new concepts have brought something different and evolving. Journalist’s Resource has put together a roundup on those significant research papers in the past : MOOCs and online learning: Research roundup.
A 2012 survey indicated that 41% of those studying online were working professionals, while 31% were undergraduates and graduates. Nearly 40% of respondents reported enrolling because of casual subject interest, and completion rates are low, hovering around 10%, as students struggle to remain motivated in an online environment. Some educators see MOOCs as the next step in the development of technology-enhanced learning practices, including blended learning and online learning portals such as Khan Academy, YouTubeU and TedEd.
Taking online learning as a great opportunity to enhance personalized learning is a good idea. But what works and what doesn’t? There are 19 papers worth to review.
“Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies”
Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. 2010. Technical Report. U.S. Department of Education, Washington, D.C.
Excerpt: “A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning…. The meta-analysis found that, on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes — measured as the difference between treatment and control means, divided by the pooled standard deviation — was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K-12 students. In light of this small corpus, caution is required in generalizing to the K-12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education).”
“Effectiveness of Fully Online Courses for College Students: Response to a Department of Education Meta-Analysis”
Community College Research Center, Teachers College, Columbia University. June 2011.
Excerpt: “Proponents of postsecondary online education were buoyed by a 2009 meta-analysis sponsored by the U.S. Department of Education suggesting that student learning outcomes in online courses are superior to those in traditional face-to-face courses. This paper demonstrates that this finding does not hold for the studies included in the meta-analysis that pertain to fully online, semester-length college courses; among these studies, there is no trend in favor of the online course mode. Furthermore, the authors argue that because these studies examine courses that were taken by relatively well-prepared university students, their results may not generalize to traditionally underserved populations. The authors conclude that while advocates argue that online learning is a promising means to increase access and improve student progression through college, the Department of Education report does not provide evidence that fully online college courses produce superior learning outcomes, particularly among low-income and academically underprepared students.”
“A Pedagogy of Abundance or a Pedagogy to Support Human Beings? Participant Support on Massive Open Online Courses”
International Review of Research in Open and Distance Learning, November 2011, Vol. 12, No. 7, 74-93.
Abstract: “This paper examines how emergent technologies could influence the design of learning environment. It will pay particular attention o the roles of educators and learners in creating networked learning experiences on massive open online courses (MOOCs). The research shows that it is possible to move from a pedagogy of abundance to a pedagogy that supports human being in their learning through the active creation of resources and learning places by both the learners and course facilitators. This pedagogy is based on the building of connections, collaborations, and the exchange of resources between people, the building of a community of learners, and the harnessing of information flows on networks. This resonates with the notion of emergent learning as learning in which actors and system co-evolve within a MOOC and where the level of presence of actors on the MOOC influences learning outcomes.”
“Why Some Distance Education Programs Fail While Others Succeed in a Global Environment”
The Internet and Higher Education, June 2010, Vol. 13, No. 3, 141-147. doi: http://dx.doi.org/10.1016/j.iheduc.2009.07.001.
Abstract: “Many universities increase their recruiting efforts to reach a larger and more diverse audience. Some universities also extend their reach with cross-border initiatives and seek international students in order to promote enrollment growth and global learning. The economic potential of distance education and academic globalization has attracted numerous higher education providers, many of which operate on a for-profit basis. The result is an increase in competition for students, which leads to added pressure on universities to control costs and rising tuition. Those online programs unable to successfully adapt to this competitive environment are at risk of failing. This article draws from the research literature and US Securities and Exchange Commission (SEC) filings to examine seven important factors that help determine the success or failure of online programs. These factors are planning, marketing and recruitment, financial management, quality assurance, student retention, faculty development, and online course design and pedagogy.”
“Can MOOCs and Existing E-Learning Efficiency Paradigms Help Reduce College Costs?”
George Mason University School of Public Policy working paper, June 2012.
Abstract: “E-Learning enrollments in post-secondary education are growing significantly but the annual cost of tuition continues to increase more than inflation. Does this mean that E-Learning cannot reduce costs? After describing some salient details of the tuition cost problem, this article examines four paradigms that have the potential to reduce tuition significantly over the long term, perhaps by half. First, it reviews the potentially game-changing effects of free Massively Open On-line Courses (MOOCs) and the new MIT/Harvard Ed X project. Both are very new and eventually target as many as a billion on-line students. Second, it describes the underutilized but highly scalable National Center for Academic Transformation’s NCAT) course redesign paradigm, capable of reducing aggregate administrative and teaching costs by billions annually, and examines problems associated with its wider deployment. Third, Western Governors University’s competency-based approach is presented as another way of drastically reducing labor costs while increasing availability of college courses to low income students. Finally, the export/import approach is introduced, in which MOOCs for credit become available (at a fee) beyond the offering institutions, thereby setting up a disruptive, but potentially highly economical reduction in faculty and administrative labor expense.”
“The Retention Fallacy”
National Center for Academic Transformation, July 2010.
Findings: “Sometimes completion rates [for redesigned online courses] will go down for good reasons. We have frequently experienced the phenomenon of improved student learning outcomes supported by clear assessment data coupled with decreased completion rates. This phenomenon is typically due to prior grade inflation… There is a relationship among section size, pass rate improvements and possible cost reduction. The larger the section size, the higher the pass rate improvement must be in order to reduce costs. The smaller the section size, the lower the pass rate improvement must be in order to reduce costs. However, reductions in smaller section sizes result in lower amounts of the cost reduction…. even if the number of students enrolled in the course is large and you take the necessary steps to reduce the number of sections offered, the impact on cost reduction may be quite small.”
“Understanding the Implications of Online Learning for Educational Productivity”(PDF)
Department of Education, Office of Educational Technology, January 2012.
Excerpt: “While rigorously researched models are lacking, the review of the available literature suggested nine applications of online learning that are seen as possible pathways to improved productivity: 1) Broadening access in ways that dramatically reduce the cost of providing access to quality educational resources and experiences, particularly for students in remote locations or other situations where challenges such as low student enrollments make the traditional school model impractical; 2) Engaging students in active learning with instructional materials and access to a wealth of resources that can facilitate the adoption of research-based principles and best practices from the learning sciences, an application that might improve student outcomes without substantially increasing costs; 3) Individualizing and differentiating instruction based on student performance on diagnostic assessments and preferred pace of learning, thereby improving the efficiency with which students move through a learning progression; 4) Personalizing learning by building on student interests, which can result in increased student motivation, time on task and ultimately better learning outcomes; 5) Making better use of teacher and student time by automating routine tasks and enabling teacher time to focus on high-value activities; 6) Increasing the rate of student learning by increasing motivation and helping students grasp concepts and demonstrate competency more efficiently; 7) Reducing school-based facilities costs by leveraging home and community spaces in addition to traditional school buildings; 8) Reducing salary costs by transferring some educational activities to computers, by increasing teacher-student ratios or by otherwise redesigning processes that allow for more effective use of teacher time; and 9) Realizing opportunities for economies of scale through reuse of materials and their large-scale distribution.”
International Review of Research in Open and Distance Learning, March 2011, Vol. 12, No. 3, 40-59. doi: http://hdl.handle.net/10515/sy5hh6ck2.
Findings: “In this paper, we have described the unprecedented affordances that Web 2.0 offers for interaction and communication and for emergent learning, as well as some of the substantial challenges in realising this potential in education… Such a framework would be based first on the technical or infrastructural conditions for emergent learning. ICT is fast morphing into the social software of Web 2.0 and the augmented reality of cloud-based Web 3.0. ICT has changed beyond recognition, providing global open access at extremely low cost, for not only consuming, producing, and distributing texts and artefacts but for interaction, communication, and networking. Secondly, emergence requires new institutional and social memes and structures. Some innovative legal frameworks which are already in place provide clear protocols and resources for collaboration and sharing, notably Open Source licences for collaborative software and Creative Commons licences for collaborative and shared content. Many free (mostly advertising-driven) platforms are also in place, from Google to a range of social software and cloud-based “apps” — downloadable applications. And thirdly, there is a need for a shift from a monolithic learning environment in which everything must be controlled and predictable to a more pluralistic learning ecology in which both prescriptive and emergent application domains and modes of learning have their place, and in which it is possible to celebrate the unpredictable. This requires quite a different mindset, in which there is a role for safe/fail as well as fail-safe management, a role for resilience as well as robustness, and a balance between pro-spective and retro-spective sense-making of teaching and learning (see section on Managing Emergence).”
“What 40 Years of Research Says About the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study”
Review of Educational Research, March 2011, Vol. 81, No. 1, 4-28. doi:10.3102/0034654310393361.
Excerpt: “The average student in a classroom where technology is used will perform 12 percentile points higher than the average student in the traditional setting that does not use technology to enhance the learning process. It is important to note that these average effects must be interpreted cautiously because of the wide variability that surrounds them. We interpret this to mean that other factors, not identified in previous meta-analyses or in this summary, may account for this variability. We support Clark’s (1983, 1994) view that technology serves at the pleasure of instructional design, pedagogical approaches, and teacher practices and generally agree with the view of Ross, Morrison, and Lowther (2010) that “educational technology is not a homogeneous ‘intervention’ but a broad variety of modalities, tools, and strategies for learning. Its effectiveness, therefore, depends on how well it helps teachers and students achieve the desired instructional goals” (p. 19). Thus, it is arguable that it is aspects of the goals of instruction, pedagogy, teacher effectiveness, subject matter, age level, fidelity of technology implementation, and possibly other factors that may represent more powerful influences on effect sizes than the nature of the technology intervention. It is incumbent on future researchers and primary meta-analyses to help sort out these nuances, so that computers will be used as effectively as possible to support the aims of instruction… there is the suggestion that one of technology’s main strengths may lie in supporting students’ efforts to achieve rather than acting ‘/.as a tool for delivering content.”
“Class Differences: Online Education in the United States, 2010”
Babson Survey Research Group, Babson College, November 2010.
Excerpt: “The first study in this series found that a majority of chief academic officers rated the learning outcomes for online education “as good as or better” than those for face-to-face instruction, but a sizable minority considered online to be inferior. Each subsequent year’s report displayed similar results. Do academic leaders still hold the same opinion, given the rapid growth in the numbers of online students?… The 2010 results show some small improvements in the perception of the relative quality of online instruction as compared to face-to-face. In the first report of this series in 2003, 57% of academic leaders rated the learning outcomes in online education as the same or superior to those in face-to-face. That number is now 66%, a small but noteworthy increase. Over three-quarters of academic leaders at public institutions report that online is as good as or better than face-to-face instruction (compared to only 55.4% of private nonprofits and 67.0% of for-profits.)”
“Conceptions of and Approaches to Learning Through Online Peer Assessment”
Learning and Instruction, February 2010,Vol. 20, No. 1, 72-83. doi: http://dx.doi.org/10.1016/j.learninstruc.2009.01.003.
Abstract: The present study investigated junior college students’ conceptions of and approaches to learning via online peer assessment (PA) using a phenomenographic approach. Participants were 163 college students. Students were asked to accomplish a given learning task via an online PA system. Of the participants, 62 were interviewed after the activity. The interviews revealed hierarchically related and qualitatively different categories of conceptions and approaches to learning via online PA. The main and achieved levels of conceptions of and approaches to learning were determined. The results showed that, within each level, conceptions emphasizing a fragmented and cohesive learning tended to be associated with approaches focusing on surface and deep learning, respectively. In addition, students with cohesive learning conceptions and deep learning approaches were likely to make greater progress in the early stages of online PA activity. The present study finally found that approaches to learning via online PA were less related to the learning outcomes than conceptions of learning.”
“Current Status of Research on Online Learning in Postsecondary Education”
Ithaka S&R research group, May 2012.
Excerpt: “Students who took all or part of their class online performed modestly better, on average, than did those taking the course through traditional, face-to-face instruction. Although both purely online instruction and hybrid forms of instruction had positive effects, the hybrid mode had a larger advantage — which is hardly surprising since hybrid forms of instruction involve the commitment of more resources. Time-on-task was found to be a significant variable, and effect sizes were larger when a new instructional approach was tried, as opposed to simply varying the medium of delivery. Significantly, the meta-analysis found that online learning can be enhanced by giving learners control of their interactions with delivery mechanisms and prompting learner reflection.”
“Motivation in Online Learning: Testing a Model of Self-Determination Theory”
Computers in Human Behavior, July 2010, Vol. 26, No. 4, 741-752. doi: http://dx.doi.org/10.1016/j.chb.2010.01.011.
Findings: “This study found a mediating effect of need satisfaction between contextual support and motivation/self-determination. In other words, supports of autonomy and competency positively affected online students’ perceived autonomy, relatedness, and competency, the satisfaction of the three basic needs. Students’ need satisfaction, in turn, positively affected online students’ self-determination… haphazard and aimless supports without addressing students’ needs are likely to lead to adverse — even worse than “no effects” — outcomes. It is through the enhancement of students’ perceptions of autonomy, relatedness, and competency that makes contextual support effective and meaningful to online students.”
”Student Interactions in Online Discussion Forum: Empirical Research from ‘Media Richness Theory’ Perspective” (PDF)
Journal of Interactive Online Learning, Spring 2010, Vol. 9, No. 1, 22pp.
Findings: “The present study is significant in its attempts to draw on different theories to investigate the antecedents and outcomes of using ODFs along with classroom lectures. The findings indicate that facilitating discourse, reflective thinking, assessment and connectedness contribute to interactions in ODFs. From the practical perspective, the present study suggests that using multiple medium of instruction enriches the communication context and leads to enhanced learning. Some limitation of this study may warrant further consideration in future research. First, the regression results must be interpreted with caution. It is advisable to use more powerful statistical approaches like structural equation modeling or experimental design to enable the testing of relationships among the constructs. Second, instructional tools other than ODF could be investigated and compared with classroom lectures for their perceived effectiveness in meeting the learning objectives. Further, qualitative methods, as used in previous research are needed to analyze the content of the postings and make inferences about the level of reflective thinking, and sense of community among the forum members.”
“Anxiety and Attitude of Graduate Students in On-Campus vs. Online Statistics Courses” (PDF)
Journal of Statistics Education, 2010, Vol. 18, No. 1. doi: www.amstat.org/publications/jse/v18n1/devaney.pdf
Abstract: “This study compared levels of statistics anxiety and attitude toward statistics for graduate students in on-campus and online statistics courses. The Survey of Attitudes Toward Statistics and three subscales of the Statistics Anxiety Rating Scale were administered at the beginning and end of graduate level educational statistic courses. Significant effects were observed for two anxiety scales (Interpretation and Test and Class Anxiety) and two attitude scales (Affect and Difficulty). Observed decreases in anxiety and increases in attitudes by online students offer encouragement to faculty that materials and techniques can be used to reduce anxiety and hopefully enhance learning within online statistics courses.”
“To Blog or Not to Blog: Student Perceptions of Blog Effectiveness for Learning in a College-Level Course”
The Internet and Higher Education, December 2010, Vol. 13, No. 4, pp. 206-213. doi: http://dx.doi.org/10.1016/j.iheduc.2010.04.001.
Abstract: “Blogs have the potential to increase reflection, sense of community and collaboration in undergraduate classrooms. Studies of their effectiveness are still limited. The purpose of this study was to investigate whether the use of blogs in a large lecture class would enhance students’ perceived learning. Students in an undergraduate nutrition course were required to engage in blog conversations over the course of the semester to promote reflective learning. Sixty-seven undergraduates responded to a survey with dimensions on perceived learning and sense of community. Sense of community and computer expertise were identified as significant predictors of perceived learning, when controlled for age, gender, and previous blogging experience. While a majority of the students reported that blogging enhanced their learning and led them to think about course concepts outside the classroom, fewer perceived value in peer comments. Implications for integrating blogging into undergraduate classrooms are discussed.”
“Procrastination, Participation, and Performance in Online Learning Environments”
Computers & Education, January 2011, Vol. 56, No. 1, 243-252. doi: http://dx.doi.org/10.1016/j.compedu.2010.07.025.
Abstract: “The present study focuses on a specific learner characteristic in the management of time — procrastination — and its role in an online learning environment. More specifically, it was expected that procrastination would influence the successfulness of online learning and that this could be explained by the level of participation of learners in discussion forums. A study was conducted to test this hypothesis among a sample of learners taking a 10-week course on environmental and land use issues. As predicted, a negative relationship was found between procrastination and performance, and this relationship was mediated by the level of the learners’ participation in discussion forums. In other words, it appears that if high procrastinators are less successful online learners than low procrastinators, it is partly due to their lack of participation in discussion forums during the learning process. Additionally, some behavioral differences between high and low procrastinators were found in the times they decided to (re)start working at a distance, felt motivated to work on their course, and felt like dropping out of the course. To conclude, some practical implications for tutoring online activities and for stimulating participation in online learning environments have been proposed.”
“Impact of Podcasting on Student Motivation in the Online Learning Environment”
Computers & Education, September 2010, Vol. 55, No. 2, 714-722. doi: http://dx.doi.org/10.1016/j.compedu.2010.03.004.
Abstract: “Researchers investigated the impact of podcasting on student motivation in the online environment during fall 2008 and spring 2009. Data were collected from students enrolled in fourteen online courses at a research university in the United States. One hundred and ninety-one students completed a modified version of the Instructional Materials Motivation Survey; it has four subscales: attention, relevance, confidence, and satisfaction. Strong positive relationships between all subscales were detected. Results indicate students were moderately motivated by the use of podcasts in their online courses. Statistically significant differences in student motivation based on gender, class standing, and prior online learning experience were found. Benefits of using podcasts and recommendations for improvement of the multimedia files were offered by users.”
Read the full post here.
About Journalist’s Resource
The site is run by staff and graduate students at Harvard Kennedy School’s Joan Shorenstein Center on the Press, Politics and Public Policy, as part of the Carnegie-Knight Initiative on the Future of Journalism Education. We invite all those interested in policy and public affairs to use the site’s materials. No registration is required; the materials are free and are under a Creative Commons license. Our open-access project is designed to provide state-of-knowledge information on topics of public interest.
photo credit: AJC1 via photopin cc