Activating Learning Methods and Learning Analytics

In modern pedagogy, we use activating teaching methods that immerse the learners into the learning experience through various means. We engage them in activities based on problem-based learning, experiential learning and drama pedagogy. Teacher’s role is no longer that of an information provider or a fountain of knowledge, but that of a motivator, supporter and mentor.

The goal of learning analytics is to provide the teacher with data that supports the teacher in guiding and supporting their students in online learning activities. At the barest minimum, a teacher will be able to see the login times of all learners, the resources that they have accessed in the learning environment and the forum discussions that they have followed or participated in. Learning environments like Moodle provide all these as well others, including a so-called “heat map” to show the teacher the most actively accessed resources or activities in the learning environment.

The central question from the pedagogical perspective is whether the above kind of data and their visualisations offer a useful resource for teachers and how this data can be used to improve learning or the guidance and support that teachers provide their students.

In the APOA (Learning Analytics in the Universities of Applied Sciences) project, several pilot courses and projects were organised independently of each other, led by various teachers with their own student groups. While some worked in classrooms or face-to-face settings and used activity rings to observe their students’ stress levels during different activities, others worked in distance learning contexts. In this paper, I will discuss a pilot, which focussed on an online course I taught to my group of professional teacher students at HAMK University of Applied Sciences, Professional Teacher Education Unit.

Pedagogical Approach in Piloting Learning Analytics and Activating Learning Methods

Distance education used to refer to a learning method where the students worked individually with the help of materials and assignments provided by the teacher and then submitted their work for assessment. This approach relied on the idea of cognitive learning theory where learners are seen as information processors and learning is viewed as an individual process of knowledge construction (Munyoro, Mukherjee, & Susimetsä, 2019). In today’s context with online tools, distance education has moved away from these traditions and the aim is to follow more modern learning theories, such as social constructive and sociocultural learning theories. We see learners first and foremost as social creatures and learning as collaboration where people construct new knowledge and understanding together (see Gbadegeshin, Appiah, Boateng, & Susimetsä, 2019). Attending an education programme is not only about learning the skills and knowledge that are needed in a particular field, but it also includes learning a new culture – the work culture and the way language is used in that field. In short, learning is as much about acculturation into a new field as it is about acquiring skills and knowledge.

In the case of professional teacher studies at Häme University of Applied Sciences, teaching is based on collaborative learning methods where learners build knowledge and understanding together, sharing ideas and points of view with each other. These are the same methods that graduating teachers are expected to use in their own future work. Different groups follow slightly different principles but, in the group participating in this pilot, the pedagogical method was based on learning by teaching. That is, the group had been divided into four smaller peer groups and they were all, in their turn, responsible for designing and carrying out contact sessions and any assignments and assessment that went with them for the rest of the groups. I supported them in the process, making sure that the contact sessions included as little lecturing as possible and as many activating teaching methods and discussions as possible (following the principles of flipped learning).

The learning analytics pilot was conducted during the last course before graduation: Networks in Education. The assessment criteria for the course in the 2019 professional teacher education programme curriculum were:

• The teacher student understands the professional role of a teacher as a developer of the workplace and as a contributor to regional development.
• They recognise the opportunities for cooperation offered by different education sector interfaces.
• They know of various national and international vocational education cooperation networks with employers and other partners and their possible uses.

In the pilot, these were broken up into two separate learning assignments. The first one was a more traditional assignment addressing the above criteria on a general level in the contexts of the learner’s own work or teacher training environment. In short, the learners read the course materials and wrote a reflective text on their own existing professional and education networks and their development needs. Following the idea of flipped learning, this guided the learners to read up on the theoretical material and reflect on it before the contact session. The second assignment was an activating group task, based on gamification and drama pedagogical approach.

Tabletop role-playing games, as they are used in teaching, belong to an overall approach known as drama pedagogy. Drama is commonly used to immerse learners in imagined situations, such as customer service or emergency response events, and it allows them to act and play through them without risking themselves or others. It is based on the sociocultural learning theory, allowing the learners to actively examine and create culture (e.g. Heikkinen, 2010). Tabletop role-playing games introduce to this the element of gamification: players play the storyline through their characters’ eyes and they use these characters’ skills (through dice rolls) to solve the problems that they face.

In the pilot, the tabletop role-playing scenario, or storyline, introduced the learners to both inter-organisational networking and education export. The narrative described a group of Finnish education specialists (played by the learners) from various universities and schools travelling to a fictional developing country to export Finnish educational thinking and practices. The goal was to use this storyline to present various ways in which networking and collaboration helps teachers in solving problems and coping with unfamiliar situations (for more detail on the game, gamification and the students’ experiences, see Susimetsä, 2020).

Description of the Learning Environment

Viberg, Hatakka, Bälter and Mavroudi (2018) performed a meta-analysis of the use of learning analytics in higher education and found that 35% of research papers had shown evidence that learning analytics improves learning support and teaching in higher education. Only 9% of the papers showed evidence of improved learning outcomes (ibid. p. 102). Similarly, Wilson, Watson, Thompson, Drew and Doyle (2017) point out that empirical research showing learning benefits from the use of analytics data is mixed at best and it runs the danger of making teachers view learning as behaviouristic process where students learn by performing specific steps or showing a certain degree of activity in the learning environment. They propose that this runs against the modern pedagogical thinking of social constructivism.

Therefore, dropping aside expectations of increased learning, the pilot course aimed to improve the timeliness of teacher’s support for learning as well as to conduct dialogical activity analysis of the learning game. The analytics tools offered in the Moodle learning environment were used to follow student activity on the first individual assignment. This mainly involved keeping an eye on the number of accesses to the material package offered on Moodle (mostly links to online sources) as well as logins to the environment and visits to specific discussion forums.

For the role-playing game activity, the participants were required to familiarise themselves with the game rules, create their characters and read the story introduction. These steps were also carried out in Moodle. For the game itself, the first intention was to use MS Teams, more specifically the Chat tool, as the platform. The purpose was to examine student activity through their inputs and reactions to the story, their involvement in the discussion etc. However, at the last moment, it turned out that the learning analytics tools on Teams could collect data only from Teams classrooms and not the Chat tool. A new solution was needed and, in the end, the role-play sessions were conducted over a Zoom voice connection and, as the research participants had not been informed of a possible audio recording of the session and such last-minute changes were deemed potentially disruptive, no data could be gathered from them (however, a qualitative assessment of the gamification experience is the topic of another article – see Susimetsä, 2020).

As a result of this switch, analytics data was gathered only on the Moodle platform. It focussed on the individual assignment and the preparation tasks for the role-playing session and was used to ascertain that the learners were designing their characters and familiarising themselves with the learning materials in timely fashion before the game session – and to send reminders and clarifications where needed.

Data on Student Activity

The participating students’ activity on the Moodle learning platform was observed with various learning analytics tools. A discussion forum was created for the individual assignment on personal networks. This forum turned out to be very active even though the assignment did not require the students to familiarise themselves, or comment on, other students’ submissions. The activity on this more traditional assignment was in line with the group’s general activity in more collaborative online tasks. With 9 to 76 interactions (mean 29 interactions) by each participant, it was evident that the students had previously learned the habit of learning from each other and they continued to read each other’s submissions even when the assignment did not specifically require it.

For the purposes of the tabletop role-playing game learning activity the student group was split into two groups of six students. Both groups used the same Moodle forum for the game preparation tasks. This included a character creation activity and the presentation and reactions to the game scenario (story set-up). In order to create their characters, the students needed to familiarise themselves with the game rules, discuss their choice of character with the other students and then post their character form onto the forum. For the story set-up, the students read through a short introduction to the narrative and then discussed and wrote out their character’s preparation process for the international trip that the story sent them on. The students were very active on this forum and each of the participants wrote between 4 to 9 messages over the four-week period and visited or interacted with the discussion threads 34 to 184 times (mean: 68 interactions). Compared to other courses in the module, this was clearly the most active forum.

Figure 1 shows the student activity during the course. The characters were created and the assignment on personal networks submitted around the end of April and the beginning of May. The character creation caused some extra activity during the middle part of the course – on the week before the online game session – as the character creation system required further explanation. The opening story was presented on Friday, 8^th May, and the live Zoom sessions with the RPG were organised on Tuesday, 12^th May, with separate morning and afternoon sessions. The course activity increased as the students read the story opening and discussed their preparation for the session, with the highest peak on the 11^th and 12^th of May.

In addition to this overall data, Moodle analytics tools were used to follow the activity of individual students. Whereas a teacher traditionally follows mostly forum activity, this additional data revealed students who were actively following and reading the forum posts but were not as active in writing or posting as some other students.

Conclusion

The pilot involved a very small group of students approaching the end of their studies. The teacher–student and student–student connections were already very strong as a result of the year-long teacher education programme. In addition, the students were strongly self-directed in their learning. These aspects were very evident in their activity on the first assignment where they went beyond the assignment requirements to learn from each other. Participation was easy to keep track of on basis of student activity on the forum and private e-mails requesting clarifications or further instructions. Thus, even without the analytics data, it was possible to see when motivating messages and reminders were needed.

The original idea was to play the tabletop roleplaying game in a text-based environment where it would have been possible to conduct a linguistic and activity analysis of how the participants engaged in the game through their interactions. However, as described above, the game had to be conducted over Zoom voice connection. Although this was probably a positive decision from the point of view of overall enjoyment of the game for the students, it reduced the scope of the resulting analytics data.

From the point of view of further research, it would be interesting to collect more detailed analytics data of similar tabletop role-playing game activities. Using, for example, the so-called activity rings to collect hard data on student stress levels during the activity might provide interesting insights into student experiences especially when compared to qualitative data. This remains a potential area for future research.

Although the pilot did not show a great benefit to teacher-provided guidance through the gathered data, it is the belief of the author that such data would be very useful to teachers starting to work with a new group of students, or with a larger number of students where the teacher cannot get to know each student individually. Similarly, this kind of analytics data would help in the guidance of students who have less developed self-directed learning skills.

More specifically, analytics data would be useful in the beginning phases of a teaching programme. The early phase is always the most vulnerable, as it involves peer group formation and learning the methods of study in addition to learning new content. By using analytics data, the teacher will easily recognise students who are not active on a learning platform and interacting with the other learners. It will be easier to intervene early and attempt to motivate the students and get them up to speed with the rest of the group. In HAMK teacher education, this would work as an additional tool to the already existing safety net of student peer groups, who work on assignments together and are given the responsibility to observe and assess their group work in addition to completing their learning assignments. However, the teacher will often be told relatively late if a peer group notices problems with the participation of any of their members and this may result in any intervention being too late. With analytics data, it should be possible to act sooner and alleviate the situation before it endangers the peer groups’ work and long-term collaboration.

It is not advisable to rely on analytics tools too much, however. Wilson et al. (2017, p. 25) note that relying on student activity data alone to assess engagement in a course may lead to false alerts. In their study, they noticed that many students who interacted relatively little in online learning environments still performed well in the course and vice versa. This evinces the old adage that learning can also happen outside the school environment. In the same vein, teachers need more skills and knowledge to be able to recognise the kinds of tools that can be used to follow student activity during activating learning tasks – many of which do not leave tell-tale tracks into official learning environments – and to recognise what kinds of data can be used to make valid judgements of actual student focus and learning.