'Let's show students what course evaluations are for’

Currently, all courses at Utrecht University are evaluated through the Caracal system. After each block, all students receive an email requesting them to fill in a questionnaire with dozens of questions, anonymously. These evaluations are meant to provide insight into the quality of the courses: according to students, what is going well and what could be improved? Caracal contains both closed-ended questions, in which students rate something with a grade between 1 and 5, and open-ended questions allowing them to share their opinions about the course and the teacher.

In the first part of this article, DUB showed that this approach is problematic. For example, the participation rate is meagre: on average, only 10 to 15 percent of students answer the questions. In addition, respondents often use the questionnaire to rebuke the teacher. Students don't only use the form to criticise the teacher's ability to get the content across, but also complain about how much work the course entails, the language skills required, and even the teacher’s gender and political views.

Everyone agrees that course evaluations are important. The question is how they could be improved, which is why the faculties have been conducting experiments at various levels. These initiatives are both tweaking the current system and trying out other forms of evaluation. "There is a need for increased flexibility in the implementation," says Renée Filius, head of the Education department. At the central level, the university is working on a memo that will enable faculties to realise that.

According to Filius, faculties are free to customise their course evaluations, but the university has penned a document with recommendations in collaboration with the faculties. This document says, for example, that evaluations should be used to improve education and not as part of staff policy. The workgroup also finds that there is no point in having standardised questions for the entire university. “We’re giving faculties the tools to improve course evaluations,” explains Filius. “We tell them that not every course needs to be evaluated and that it is possible to use shorter questionnaires. In addition, we note that it can be stimulating for students to evaluate a course halfway through the term as well. In this case, evaluations could be addressed immediately, which is favourable for students. Although there doesn't have to be a minimum number of participants, we also observe that it is more useful to look at the suggestions made in the open questions when the response rate is low. To prevent hurtful remarks from being widely shared, one has to take a critical look at who has access to the open questions.”

The idea is to implement the recommendations from the note on Caracal later on. This way, it would be easier for faculties to make their own choices.

Evaluation plan
One of the ways course evaluations could be improved is when faculties or study programmes present an evaluation plan. They should mention how the evaluation is being handled. Marlies van Beek, Education Advisor and trainer in the Education Advice & Training department of the Faculty of Social Sciences: “It’s not necessary to evaluate a course every year. And it’s good to point out what you will do with the information. How are teachers going to use it? How will the education coordinator or the education committee use the information to improve education? In what other ways is the quality of the course being evaluated?"

Van Beek also has suggestions on how to improve the response rate. "At the start of a lecture, you can point out what the results were last time and how they influenced the development of the current course. At the end of the term, encourage students to complete the course evaluation and indicate what is expected of them. That way, you get students more engaged and you can prevent them from making hurtful comments. In my view, course evaluations should be filled in anonymously. Students need to be able to express their opinions freely."

Psychology student Claire Bruls thinks that's a good idea. “Course evaluations were not mentioned at all in our classes. We just got an email after the course was done. As a student, I would have been more stimulated if teachers had talked about what they did about last year’s evaluation before we started.”

Thermometer
This academic year, the Faculty of Science is trying out a different evaluation format. In this project, titled The Thermometer, questionnaires were reduced to no more than five questions, three closed-ended and two open-ended. Aligning these questions across the faculty makes it easier for them to compare results. "Students are now spending less time on evaluation forms, so we hope this will increase the response rate. By briefly gauging the most important elements in a course, teachers will soon know what they need to pay more attention to. The Thermometer gives them an initial indication. The shorter version of the questionnaire does not explicitly ask about the teacher's performance," explains Susannah Standing, from the Faculty of Science.

One thing will remain the same, though: students will not be asked to evaluate the course before it ends. “Ideally, students would fill in the evaluation during the course, but we chose to include a question about the exam. But we did ask teachers to stress, throughout the course, how important it is to fill in the evaluation form.”

Fewer distressing remarks
Other faculties are also considering or working on making the questionnaire shorter. The Faculty of Humanities reduced the number of open-ended questions in September 2022, for example. The faculty's main goal was not to improve the participation rate but rather to reduce the number of unkind comments.

Previously, the faculty drew up a protocol on what to do about hurtful and intimidating remarks. If the remark in question is traceable, the education director is supposed to get in touch with the individual who made it. However, such remarks usually cannot be traced because the questionnaires are filled in anonymously. In this case, the issue is addressed in the course environment, underscoring that such comments are unacceptable.

The Faculty of Science also expects their new format to reduce the number of offensive remarks. "The questions focus more on the content of the course rather than the teacher's performance," explains Standing.

Engaging in a dialogue
Rector Henk Kummeling would like the role of students to be more emphasised in education. "It should not be a one-sided assessment, but more of a dialogue," he says.

He isn’t the only one who would like the evaluation of courses and teachers to be completely dependent on Caracal. Susannah Standing says that the Faculty of Science sees The Thermometer as one of the sources of information about how students experience courses and how courses could be improved. "We encourage programmes to see Caracal as one of the possible information sources. The Thermometer should provide an initial indication after which we can use the results to investigate things further, if necessary." 

Marije van Braak, a teacher of Communication & Information Science, sees the Caracal assessments as a mandatory exercise from which she doesn't get much useful information. "The questionnaires often contain conflicting information. Some students find the lecture too slow, while others complain that the pace is too fast. What are you going to do?" She believes that a format based on dialogue would work better. 

She teaches face-to-face communication, a course whose evaluation is made through a conversation included in the course's design. “We are doing a pilot in which students discuss their experience of the course in duos, based on associations. You can trigger associations by using colours or association cards, for example. The conversation is recorded and the playback is shared with the teacher after the exams. However, students also write a short report during the lecture about its main conclusions. This can trigger an initial conversation with the teacher. I think this form of evaluation has more value because the teacher gets more concrete tools to improve the course and the students reflect on their role in class."

Another alternative is working with panels. A student representative gathers information from fellow students, makes a summary of the input and then shares it with the teacher or the education coordinator. The Faculty of Social Sciences is considering this format. “It’s important to have student representatives that are trained to gather proper information. That’s an investment,” Marlies van Beek says.

Own questionnaire
The Psychology programme has come up with a completely different approach for first-year workgroups. A workgroup consisting of approximately twenty students gets a junior teacher as a permanent supervisor. “Students then bond with that teacher. We have our own standardised questionnaire twice a year, separate from Caracal, which we use to assess the progress. Students get about fifteen questions, some of them open-ended, so they can share their tips anonymously. How are students experiencing their year and how do they evaluate the cooperation with the teacher? This form of evaluation works well and the response rate is high,” explains Maarten van der Smagt, Educational director of the Psychology programme and supervisor of the junior teacher group.

Peer review
Another method being used here and there at UU is peer review. In 2018, DUB published an article about course evaluations that already mentioned peer review as a method. It works as follows: a teacher asks colleagues to watch classes throughout the course and then give feedback about the course's structure and the quality of teaching. 

The right method to evaluate courses is yet to be found, however. "Evaluations are important as they can be helpful to improve education," say students Claire Bruls and Teresa Pappalardo, who joined the University Council to think along about education. "But it should be clear to students why they are asked to evaluate the courses and what the university is going to do with the information collected." 

About this article
This is the second part of an article about course evaluations at Utrecht University. The first part was published in December 2023. Both articles were possible thanks to a contribution from the Fund to Stimulate Education Journalism.