Proctorio was widely used during Covid
Anti-cheating software could be racist, says human rights institute
From the left to the right: Hans de Zwart, Robin Popcornie and Naomi Appelman. Photo: courtesy of HOP.
Logging in ‘while black’ doesn't always come easy. Robin Pocornie, a student at VU Amsterdam, was sometimes not able to access the anti-cheating programme Proctorio because her face was not recognised. She suspects this might have something to do with her skin tone.
The software was widely used by Dutch universities during the coronavirus pandemic as it allowed invigilators to keep an eye on students while they took an exam from home. The software uses facial detection and Pocornie’s face could sometimes not be found, which meant that she was occasionally ejected from the exam with error messages like "face not found" and "room too dark".
Enquiry
She filed a complaint at her university, which denied that there was any discrimination going on. The university even got in touch with the supplier of the software, which said there was nothing wrong with it. The company claims to have carried out an external investigation.
VU Amsterdam says it is more likely that the student was kicked out of exams because of an unstable Internet connection, rather than the colour of her skin. After all, other students had problems logging in as well and there were no similar complaints from students of colour.
An interim decision shows that the Netherlands Institute for Human Rights does not go along with that. It is, after all, beyond dispute that the student had problems logging in. Moreover, scientific research shows that facial recognition software generally does not work so well with dark-skinned people.
Not public
In the institute's view, the university’s counterarguments do not weigh up against that, not to mention that the investigation commissioned by Proctorio was not made public. In addition, the university has not given out any information about the average login time of other students.
VU Amsterdam now has ten weeks to refute the suspicion of discrimination — otherwise, the ruling will go against the university. The decisions are not binding, however. The institute cannot impose a fine or award damages.
“I’m very happy that the law is on my side,” celebrates Pocornie in a statement to the press. She is curious to see the reaction of other higher education institutions that used the same software.
Important moment
The human rights institute is satisfied too. “Although this is only an interim decision, it is an important moment in the history of our decision-making” declares the chair of the institute, Jacobine Geel, on its website. “Someone has succeeded in saying: ‘Hey, this algorithm is doing something odd. I suspect the algorithm is discriminating.’”