After four long years, UU chooses in-house software for course evaluations

All UU students, with the exception of Medicine ones, use a computer programme called Caracal to complete their course evaluations. For a long time, it seemed like Caracal’s days were numbered, as faculties no longer wanted to be responsible for the software with which students assess their education. Instead, they would prefer the Information and Technology Services (ITS) department to take over this responsibility. That’s why, four years ago, UU began to look for a system that could be implemented throughout the entire university.

The search went on in fits and starts, however. After an initial tendering procedure, the university was keen on staying with Caracal, as it had the best test results. But that decision was later recanted because the university’s policy states that in-house software should not be used in case standard software by external parties is available. Since Caracal was developed by the Faculty of Science, it was disqualified.

A second tendering procedure ensued, but huge mistakes were made: after a winner, DiCe, was announced, it turned out that it didn’t meet the formal requirements. Yhe tendering process had to start all over again.

Software was overloaded
In 2018, a steering group comprised of representatives from the faculties, as well as the ITS and Academic Affairs departments, chose Evalytics as the evaluation system to be adopted by all UU faculties, following in on the footsteps of the universities of Nijmegen and Rotterdam, who had been using it for years.

But that didn’t work out, either. It didn’t take long for a number of issues to be reported, after the first employees started using the software (namely those working at the Faculty of Social Sciences, especially Veterinary Medicine). Evalytics simply didn’t match UU’s methods, thus requiring a lot of manual labour from teachers and supporting staff, according to ITS spokesperson Saskia van Dijk. “It wasn’t easy for teachers to add or change questions at any given moment, for example.”

The university tried to find a solution with the software’s supplier, to no avail. According to Van Dijk, it wasn’t the provider’s fault: the software was overloaded because the faculties often had varying requirements and Evalytics “couldn’t create a separate version of its software especially for the UU”.

Familiar or references
Last year, UU and Evalytics jointly decided to part ways. Since then, both Veterinary Medicine and Social Sciences have gone back to Caracal, but the question remained whether the mismatch could have been prevented. Why did no one take the time to test whether or not the software was suitable for UU’s methods?

The ITS department says an elaborate trial could have made the issues surface, but those trials are usually conductive with innovative technology, which wasn’t the case here. Rotterdam and Nijmegen had been using Evalytics for quite some time. Van Dijk: “We relied on the references we received and the questions the supplier answered.”

In consultation with the faculties, UU now decided to make an exception for Caracal, allowing it to be used as the university-wide system despite the software having been developed in-house. “It’s true that we should always research whether we can use standard software, but if it’s well-substantiated, we can definitely choose to do something different,” says the ITS spokesperson.

Competing with Caracal
Tijmen van Dobbenburgh, from Evalytics, says the end of the collaboration is “a shame”, but he understands UU’s considerations. He feels that it was hard for Evalytics to compete with Caracal because many UU employees were used to the in-house system. Additionally, because the university did not implement the connection to UU data in time, users had to add a lot of things manually, which didn’t make for a smooth experience.

What’s more, Van Dobbenburgh finds that the university lacked a clear vision of what the course evaluations should provide. “You can tell that that there are different ideas and desires in this regard.”

In need of a clear vision
That’s precisely what the university aims to do this year. In collaboration with Caracal and the ITS and Academic Affairs departments, the IT department at the Faculty of Science will develop a unified vision for the course evaluations.

Teachers have long awaited reconsiderations about this topic. Course evaluations are currently said to be a customer satisfaction study, instead of contributing to improve education. The low response rate and the amount of harsh critiques are also matters of concern.

“We’re first going to identify the vision on evaluations, the desires and needs regarding them, in collaboration with the faculties”, explains ICT Department Manager Marjon Engelbarts. Three years ago, she lamented the university’s decision to stop using Caracal, so she’s happy to contribute to a solution, she wrote in an e-mail to DUB.

Engelbarts says the Caracal version that’s currently in use is sufficiently stable – however, if UU intends to continue using it in the future, it does need to invest in the software so that it can be developed further.

So, after four years, we’ve come full circle. A lot of time, energy and money has been wasted in the meantime and it’ll take some time before the long desired university-wide evaluation system is ready to go. The departments of ITS and Academic Affairs acknowledge that the way things have been conducted was not the best, in hindsight. Saskia van Dijk: “In those instances, you need to choose not to hold on to that course at any cost, but rather decide to reconsider and choose a new direction.”