EMERGING DIALOGUES IN ASSESSMENT

“After a bit of deep thinking…”: Using Student Reflections to Assess Student Learning¹

August 15, 2023

Shauna Wilton, Ph.D and Mélanie Méthot, Ph.D, University of Alberta

This article discusses an example of how student reflections can be used for assessment, the advantages and pitfalls of using reflections for assessment, and how we used the process to improve our teaching and students’ learning outcomes, thereby closing the assessment loop. We argue that self-reflection is an important part of student learning and can be used effectively to assess learning and teaching.

Students often have flawed conceptions of their own skills. They can both over and underestimate their abilities. The Dunning-Kruger effect argues that ignorance of our ignorance can lead to inflated ideas of our expertise (Dunning 2011). In other words, being aware of what we do not know allows us to properly assess our skills. Asking students to reflect on their academic skills at different points in a course encourages them to consider skill development as a valid learning objective. It offers them the opportunity to assess their skill levels, their development, and how they learn. Reflections improve metacognition (or thinking about thinking and learning) and confidence by making students aware of what they are learning, how they are learning it, and their mastery of the course material and skills. It also provides professors with valuable data to develop assignments tailored to the needs of their students.

In an interdisciplinary first year class, we asked students to reflect on their skills as critical thinkers, researchers, and communicators (the core skills on our campus) at the beginning and the conclusion of the course. The first reflection focused on student perceptions of their own skills and what they hoped to achieve in the course. For the final reflection, students were encouraged to consult the original reflection and comment on what had changed (or not), how it had changed, and areas needing further improvement. The assignments encouraged students to consider what they know and do not know, what they need or want to know, the learning that occurred, and, last but not least, to take agency and responsibility over their learning.

The course focused on developing student skills as thinkers, researchers, and communicators, and preparing them for future research and dissemination in their senior classes. Each assignment was designed to enable students to practice and to develop their skills in the three areas. As Scholarship of Teaching and Learning (SoTL) researchers, we were interested in how the course impacted student metacognition. Furthermore, we wanted to develop ways to use the reflections in our program assessments, both to provide a baseline as they move through the degree and to determine whether we met the course learning objectives. As the instructors, we sought answers to three related questions. First, does the practice of student reflection foster metacognition and the student’s ability to think? Second, how can we improve the use of reflections in our classes to deepen student learning? Third, how can the reflections be used to inform and improve our teaching?

Many authors argue that self-reflection supports learning, and that deep learning is more closely associated with deep reflection. Harvey, Coulson and McMaugh (2016) rightly note that the evidence on the impact of reflective practices on learning is “primarily through anecdotal, practice-based evidence, student self-reporting and small case-based studies.” Furthermore, they suggest that there is a “tendency amongst researchers in the field to write with the assumption that reflection “works”, without providing evidence that this is so” (2016, 1). Our goal was to find concrete evidence on the effectiveness (or lack thereof) of reflections and their appropriateness as a tool of assessment.

To understand how the students thought about their skills, we collected the initial and final reflections from all consenting students in four course sections taught over two-years.² In the end, we analyzed 91 initial reflections and 88 final reflections using a grounded theory coding approach (Urquhart, 2019) to identify key themes. Each reflection was coded by three individuals. Any discrepancies were discussed and reconciled.
Our analysis found that the use of reflections helped students identify specific skills, increase critical thinking, and develop confidence in their skills. Students identified more academic skills in the final reflections and tended to focus on higher level skills. For example, in the initial reflections, the most common skills mentioned were critical thinking (73.6%), gaining disciplinary knowledge (65.9%), life skills such as managing workload and time (63.7%), collaboration (63.7%), and learning to do research (63.7%). In the final reflections, the most common skills mentioned were critical thinking (80.7%), written communication (68.2%), life skills (62.5%), presentation skills (62.5%), and information literacy (62.5%). Thus, the skills in the final reflections were more focused on the core skills we were teaching and demonstrated that the students improved their vocabulary and understanding of these skills.

Our analysis confirmed that the reflections were useful for understanding student knowledge of skills and their own perceptions of their abilities. It did not, however, address whether their perceptions matched their skills as demonstrated within the reflections. For example, did their written communication improve between the first and second reflection? Did they demonstrate critical thinking in their writing? Could the assignments be used to assess not only student metacognition but also the academic skills being taught in the course? The analysis also demonstrated that we needed to develop clearer questions and instructions for the rubric to help the students. As professors, we often assume that students know how to do a scholarly reflection, but our evidence showed that is not the case.

These questions led to the development of a new rubric that could analyze both what was mentioned and how it was presented in the reflections (see attached rubric). The rubric and grading of the reflections will provide the data for the assessment of student learning, reducing the amount of work required for learning outcome assessment. We also used the assessment findings to inform our teaching, in particular influencing the way we introduce and structure the assignments, attempted to reduce the differences between sections of the course, and focused our teaching more on those skills we wanted students to develop. Realizing that students do not always pay attention to rubrics, we spent more time discussing the rubric and assignment goals with the students.

In 2022, we once again engaged in an analysis of student reflections completed since the implementation of the new rubric and associated changes. While more analysis of the data is needed, it appears that we have been modestly successful. Students tended to identify more skills in the final reflections in the 2022 analysis, and the distribution curve of the number of skills identified in reflections shifted higher in the final reflection. As well, the average grade for each category increased between the initial and final reflections. 

In conclusion, formal reflections play a part in many classes, but their usefulness for improving and assessing student learning is understudied. Our multi-stage process began with the use of reflections and an initial assessment of them. We then used our findings to change the way we taught the course, introduced, and assessed the reflection. Our initial analysis of the second round (or loop) of reflection data shows that these changes led to some improvements. Ultimately, our data suggests that using rubrics to analyze not just the content of the reflections, but rather the skills demonstrated within the reflections, enables us to draw broader evidence-based conclusions about the learning within the course and student improvement.

Endnotes

¹ The quotation is taken from a student’s reflection. We wish to acknowledge the research assistance of undergraduates Frédérique Heinrich and Tyler Weenink as well as the funding provided by the Augustana Research Funds for their research assistance.

² This study received ethics approval from the University of Alberta. Consent was obtained from all student participants and a research assistant removed the work of students who did not consent and anonymized all the assignments.