Learning Analytics Framework for Multiliteracies Assessment (#learninganalytics)
Evidence-centered design suggests that any system of assessment begins with a conception of the capabilities one wants to develop in a learner; from there develops the machinery of the assessment system, such as tasks, rubrics, and scoring models (Messick 1994).
From “Analytics to Literacies: The Development of a Learning Analytics Framework for Multiliteracies Assessment“(Shane Dawson and George Siemens):
Learning Analytics methods such as social network analysis(SNA), epistemic network analysis (Shaffer et al., 2009), and affective learner attributes (Baker, D’Mello, Rodrigo, & Graesser, 2010) merged with computational linguistics moves the focus of learning analytics from the measurement of an end product towards an evaluation of the process of learning.
It is unlikely such a model can be developed through a reliance on extracting student assessment data and LMS activity alone. Grades and LMS activity do not sufficiently represent the diversity of social and cultural based interactions students frequently engage in as learning is not the sole domain of formal institutions. This calls for an examination of how student engagement across multiple educational and social systems can be captured and incorporated, including the more qualitative artifacts such as student discussion postings, essay writing, blog posts, YouTube, Facebook or Twitter feeds. The inclusion of social network methodologies provides a rich stream of data and an important analytical view to determine the types of relationships and extent of participation in defined communities. In this context, analytics can begin to capture the necessary insights that relate the individual to the community – or in the terms of Jenkins, the individual’s active engagement in a participatory culture. The following section outlines possible metrics for evaluating multiliteracies.
As with any kind of evaluative measurement, defining an outcome or process’s success indicators will guide the selection of tools for assessment. For the purposes of this paper, Jenkins et al.’s proposed new media literacies and definitions (Table 1) will be used as the basis for the learning analytics framework. Jenkins et al.’s multiliteracies have been clustered in order to refine the types of analytical data that can provide sufficient lead indicators of competency.
For example, play, performance, and simulation are closely linked in terms of their affinity with problem solving processes, experimentation, and risk taking. This may be realized through activities such as gaming or role play. However, simulation also corresponds to aspects of appropriation, whereby these skills involve a form of creation or co-creation. Items such as collective intelligence, judgment, and negotiation relate to accessing, sharing, and evaluating information and resources within and across networks. The skill of distributed cognition lies in the intersection between accessing and sharing information and navigation and multi-tasking. Figure 1 presents a visualization of the associated grouping of Jenkins et al.’s (2006) classification of media literacies. The four clusters described above can be measured, monitored, and reported through a diversity of analytics and modified for the specific pedagogical context.
Mapping the proposed four multiliteracies (see Figure 1), experimentation, products/creation, network agility and citizenship, and task effectiveness and efficiency, to an existing framework of learning analytics techniques and applications (Siemens, 2013) can provide the analytics opportunities as described in this paper. It’s quite a thorough summary of relevant learning analytics techniques and applications.
Multiliteracies draw attention to the skills and attributes learners require to navigate the increasingly complex technical, social, cultural, and economic worlds. This paper builds upon established theoretical models to demonstrate the role learning analytics can play in assisting educators and students in developing real-time feedback and analytics for evaluating literacies needed for this century.
Analytics to Literacies: The Development of a Learning Analytics Framework for Multiliteracies Assessment
Shane Dawson, University of South Australia, Australia
George Siemens, University of Texas-Arlington, USA