UU scientist measures effect of public activities: ‘This has to move out of the ‘hobby’ realm’

More and more attention has been paid to the importance of explaining to the general public what scientists do and why their work matters. If people who are not acquainted with scientific research get to hear about what scientists do and what their methods are, their trust and understanding of science shall increase. It is a good deal for scientists, too, as they can learn from the questions people ask.

This type of activities also fit right in with the new approach to 'recognition and rewards' adopted by Dutch universities, in which publications in esteemed journals are no longer the only thing that counts when evaluating a scientist's performance. Teaching top-notch classes and engaging the general public are just as important.

That's why plenty of public events aimed at all sorts of target audiences have been taking place in Utrecht, such as the knowledge festival Betweter, the science show Operatie Breinbreker, and the summer school Junior, aimed at secondary school students. Many UU scientists collaborate on these activities, which are organised by UU's Centre for Science and Culture (CWC).

Activities like these have also been getting financial support from a number of sources. Scientists with projects in science education or communication can apply for grants of up to 2,000 euros. One of the initiatives making use of such a grant is the ‘Climate helpdesk’, in which scientists answer questions about climate change.

But are these initiatives effective? How much do people actually learn from them? Do they cause people's attitude or behaviour to change? How can scientists and organisers known when a project is successful?

‘Things have to be more professional now’

That's what UU social scientist Madelijn Strick aims to find out. She was asked by NWO to assist researchers and event organisers in measuring the effect of their efforts. She is joined by Anne Land-Zandstra and Ward Peeters, two employees of the group led by Leiden Science Communication Professor Ionica Smeets. Peeters is also partially employed by UU.

Strick: : “We’re working on instruments to measure the impact of these events. At the same time, we’re using those instruments to study what does and doesn’t work.”

Strick is a veteran in the world of science communication. She was involved in a project aimed at raising awareness of their unconscious biases among citizens, as well as with the play Mindlab, which delves into about inappropriate behaviour at universities. Strick has recently been honoured with an award by the Royal Netherlands Academy of Arts and Sciences (KNAW) for her work in this area.

Personally, Strick prefers the term "public engagement" over "science communication", for it puts an accent on the interaction between scientists and civilians. Her view has also been embraced by UU. The university has recently come up with a Public Engagement plan (accessible with Solis ID only) and made Public Engagement one of the foundations of the Open Science programme.

The researcher emphasises the importance of thorough evaluations of public activities. “This has been stuck in the realm of hobbyism for a long time. Researchers enjoyed doing it so they did it on the side, without getting much recognition or compensation for their time. We’ve now come to a point where things have to be done more professionally.”

We’re looking for patterns: what works in what setting?

Strick says that, in the field of social sciences, a lot of experience has been gained with measuring instruments, methods, and questionnaires to analyse attitudes and behaviour. “I found out that they’re very appropriate to measure the effect of public engagement. It is, of course, tricky to prove that an activity leads to long-term societal change, but you can see whether or not the right effect has been accomplished.”

Strick and her colleagues have since compiled a toolbox containing approximately 30 instruments for scientists and support staff to evaluate their activities. Examples include post-its to put on walls, (video) diaries to maintain, and mind maps to create. “Public activities can be so diverse. The target audience is always different, as is the location, the specific goals they aim for… that demands a variety of options to choose from.”

The three are also working on a ‘base instrument’, a short survey that can be conducted in all kinds of activities. The survey is currently being valid – a process that includes presenting the survey as part of the public activities organised by the 90 scientists who, like Strict, received a KNAW award.

Strick: “Some scientists just want a simple template. The benefit for us is that, if many scientists start using this survey, we can do meta analyses. Based on the data we get, we can look for patterns: what works and what doesn’t? For which target audience? In which setting? But if a researcher or supporting employee wants to know something specific about what the audience learned from the explanation of mitochondria or whatever, they should use something from our toolbox.”

We’ll be able to provide researchers with advice that’s been better substantiated

Stephanie Helfferich, Public Engagement Project Leader at the CWC, says the instruments are very welcome. “We’re constantly working to improve our events. We used to base our evaluations on our own experiences and insights – our gut feeling, so to speak. But now we can be much more specific, with better substantiation, which helps when we advise the scientists who participate in our activities or organise an activity themselves.”

Helfferich has implemented some of the instruments from the toolbox in recent activities. Experience has taught that it isn’t always easy to collect information, especially when it concerns children. “The trick is to question your audience in such a way that it doesn’t harm your event, as it should remain fun and interesting. In the most recent Junior Summer School, we let children answer questions in a playful manner in the Botanical Gardens. Those questions helped us evaluate the week, but due to a lack of time in the programme, some of the children skipped that part.”

Helfferich says most of the scientists she works with understand the importance of good evaluations, even if they take precious time. “They know how much effort and energy it takes to involve the general audience in your work. So they do want to know whether it was time and effort well spent.”

It’s not like you get promoted when your audience gives you a 10

Strick warns that her measuring instruments shouldn’t just be used by managers to assess the public engagement activities of individual scientists. “I would be very much against that. These instruments are meant to evaluate activities, not people. It cannot be like 'your audience gave you a 10, so here’s your promotion'”

But scientists will have the option to indicate (also in performance reviews) how they wish to work on making their efforts more effective. “They can now show what they wanted to accomplish, how it worked, and what they learnt from it. So, it’s an evaluation cycle similar to what we use in our education.”

By the way, the social scientist does hope that the university will support employees looking to develop themselves in public engagement. She says it’s a good sign that UU has recently appointed Erik van Sebille as the first professor in the Netherlands whose task description includes public engagement.

Strick: “But not everyone can become a professor. It’d be great to have a development track just like we’ve got the basic qualification for education. It’d be interesting to think about a basic qualification for impact, and which skills would need to be a part of that.”

We’re not going to drag anyone on stage who doesn’t want to

Strick and Helfferich emphasise that the so-called outreach activities are not mandatory for all scientists. Helfferich: “We’re not going to drag someone on stage if that person doesn’t want to be there. But, for the university as a whole, and for discipline groups, it’s a task to show themselves. Managers need to offer space for the scientists who have the talent for it, and who want to do it.”

Most of all, Strick hopes that the new instruments and the research based on it will contribute to “a culture of curiosity” in the years to come. “Researchers are naturally curious, that’s the core of their work. I hope to contribute to their curiosity about how their public communication is received.”

The level can be higher in many cases’

Strick and Helfferich know many of the pitfalls scientists fall to when approaching laypeople from experience. “Too many details,” names Helfferich. Strick thinks that researchers often don’t challenge their audience enough. “They focus solely on a simple explanation of their research, but in many cases, the level could be higher.”

There’s also some harsher information. Surveys conducted by Strick with visitors of the knowledge festival Betweter show that the message comes across best when science becomes personal. Strick: “People want to understand how your research affects them. Oftentimes, the best way to achieve that is by sharing your own struggles as a students. For example, I study how to influence people with humour, but sometimes I wonder whether or not I should explain to people how they can manipulate others. If I tell a story like that, people can relate.”

Helfferich: “That’s exactly what we always try to do: connect the audience. For us, this was confirmation that we’re on the right track.”