Using Evidence Reviews for Better Decision-Making

In this episode of Pocket Change, Professor Peter Bragge explains how evidence reviews  help decision-makers gather evidence efficiently, while ensuring accuracy is not compromised. He offers some examples of how evidence has been effective and successful in solving complex societal problems, including in health and anti-social behaviour.

Grab a coffee, press play, and enjoy this episode of Pocket Change:

Pocket Change is a series of pocket-size videos about a key aspect of behaviour change.  Each episode features a BehaviourWorks Australia researcher explaining their area of expertise.  

‍

Transcript:

Geoff

Hi, everyone. Welcome back. It's 2024, and my first pocketchange guest this year is Peter Bragge. Peter Bragge, welcome.

Peter

Thank you for having me.

Geoff

Your special power, so to speak, is evidence reviews? Sotell me about evidence reviews and the importance of gathering evidence anddoing research for behavioural science.

Peter

Okay, So we live at a university and there are hundreds ofuniversities around the world, and they all do research to answer reallyimportant questions on things like climate change, sustainability, health andmedical topics. So there's a lot of activity going on around the world, andthere's huge libraries of research that have already answered a lot of thequestions that we are facing.

What happens with that research is it tends to sit on theshelf and it isn't really put to work. So my job is to get it off the shelf,turn it into a summary and hand it to someone so that they make a slightlybetter decision.

Geoff

So when you talk about a shelf in a library, you're actuallytalking about the Internet as a virtual shelf. Is it true the internet has 5 billion pages?

Peter

Yes, five plus billion pages. There's a little counter somewherein the world that says how many pages there are on the Internet. And there's alot. And then if you have a look at research as a subset of those pages, it's avery, very, very small subset.

So there's probably about 60 or 70 million research studiesin the world that are sitting on a virtual shelf somewhere. So that's a verysmall percentage of five point something billion. And then there's an evensmaller percentage of studies that have brought together what those studieshave found into things that we call research reviews. And I kind of write them.

Geoff

Give us an example of a brief. You have to find the evidenceand find it quickly.

Peter

Okay. Like all issues that are faced by governments anddecision makers and leaders, it starts with some sort of problem that has to besolved. So the first step in our process is to sit down and talk with them andwork out exactly what knowledge you need to solve this problem. So to give youan example, we got approached by the Victorian Health Promotion Foundation, whosaid we'd really like to know what works for governments to promote healthypopulations, to improve public health. Right now, as you can imagine, if youthink about all the public health messages we've been exposed to, things likesmoking, exercise, using sunscreen, and then you multiply that by all thegovernments around the world, that's a huge topic. And we had about six weeksto do that. So we said, look, we obviously can't tackle this in six weeks. It'sprobably about ten PhD's worth of work. Why don't we find five examples fromaround the world and we'll have a very quick look at the big research studiesaround those examples. And then we'll talk to one person who's been involvedwith them.

Geoff

Why is there so much pressure to get evidence quickly now?

Peter

Because people have to make decisions quickly. So, forexample, most people who do a study, the first 8 to 12 months of their isusually doing a review of the topic to work out what's the knowledge gap. Thenyou think about governments, for example, the federal government, they have athree year term. Right. And they like to get a lot of stuff done. And mostlythey have to get things done in a fairly quick way, or at least when they'remaking decisions, they have to make them fairly quickly. The reality is mostpeople who do my sort of work get weeks, not months. And sometimes it mighteven be one or two weeks. Right.

So we have to be really targeted in the question that weaddress, which is why we sit down with them and say, okay, what is it exactlythe problem that you want to solve? Because that that if you then think about 5billion pages on the Internet, that narrows the funnel quite a bit. If you knowexactly what you're looking for.

Geoff

So in the VC health example, what were the five sort offindings that you found?

Peter

The five examples were really interesting. There was oneabout food labelling laws in Chile. There was one about the so-called sugar banor the healthy food initiatives in New York, which was actually around reducingtrans fats, but also reducing sugar in takeaway foods. And then there's anotherreally interesting one in South Africa where during COVID, they completelybanned alcohol sales. There was a road safety initiative in Sweden calledVision Zero, which has been very successful, which is about changing theenvironment of the road system through things like barriers and speed humps toreduce the chance of injury.

So we had good examples covering, you know, big issues ofour time, you know, alcohol consumption, healthy diet, safety on the roads.

Geoff

So give us an example of unintended consequences withsomething like an alcohol ban.

Peter

That seems like a pretty good idea because we know thatexcessive consumption of alcohol relates to things like violence that hashealth effects. But there were some unintended consequences in terms of banningalcohol outright. So, yes, lots of lots less people went to the emergencydepartment being involved in drink driving and fighting. But there was also a bighit to the economy. Because obviously everyone who sold alcohol was effectivelyout of business, short on time.

And then there were people who were alcohol dependent whoobviously were going through a significant alcohol withdrawal. And that's notnecessarily the best way to sort of reduce your alcohol consumption, butgradual. So had some great effects, had some unintended consequences. And thenwhen COVID finished, you know, they kind of went back to business as usual.

Geoff

give us an example of a success story, a behaviour changesuccess story.

Peter

So let's think about removing trans fats in takeaway food.So trans fats, as many people would know, are kind of like the bad fats. Andthere are alternatives to using trans fats that have much less impact on thingslike heart health.

So what they did was they essentially worked with thetakeaway food industry and and the food industry to replace trans fats withother agents that had the same effect. And the beauty of something like that isit's pretty invisible to consumers because they're still buying a packet offries or whatever. You can't really taste the difference. So, highlysuccessful. And they had evidence that it reduced the amount of people whowould go into emergency department with heart attacks. Good communicationbetween government and industry here. And then a really solid measure ofsuccess in terms of the public health impact. So that was a real success story.

Geoff

In your job, in your role here at BehaviourWorks Australia,are you seeing a speed up of the pressure to come up with evidence?

Peter

We've always known that governments want answers very, veryquickly. And now we have techniques that allow us to respond to that needwithout compromising the rigour that we have associated with longer reviews.

Geoff

What is to stop a politician or a leader basically switchingon the computer and saying, Hey, Internet, how do I fix world hunger?

Peter

So you raise a very important question about the role ofartificial intelligence, in particular because I remember way back when Istarted teaching how to review evidence, I used to say to the students, you know,you can't just type in the question and get the answer, because in those daysyou actually couldn't, you had to come up with very special terms, go tospecific databases. Now we can type in a question and get an answer. If youwere to type in something about how to end world hunger, for example, you wouldcome up with some information about ways that you could do that, but it won'tbe tailored to your particular settings.

So let's say, for example, we need to provide micro-financeand invest very heavily in agriculture. What if you in a country that doesn'thave a very good environment for agriculture, or what if you don't have thecapacity to put together a micro-finance program. So the first thing aboutthose kind of global answers is that generally not tailored to where you are inthe world or what you particularly want to do. And so if you have a betterunderstanding of what the question is that you're trying to answer, you mightlook for things that are, for example, how do we solve world hunger in resourcepoor settings that have economies of a particular size? So then you're gettinga more targeted question and a more targeted answer. You're still usingartificial intelligence to sift through large amounts of information, butthey're coming back with an answer that's more likely to be relevant.

Geoff

So even with advances in AI in terms of search engines andsearch terms, a human has to pass that information to see if it's relevant.

Peter

Correct. And then the other thing that is a little bit of arisk with artificial intelligence is how do they make judgments about thequality of the research? So in traditional evidence reviewing or even in rapidreviewing, a human usually reads a study and makes some judgments about thingslike the strength of the study design or whether there were enough people inthat study or whether there might have been some bias in the way that they wererecruited. Ideally, someone somewhere has read that article and applied at all.That actually goes into a bit more depth about the quality of the research.

Geoff

Peter Bragge, thanks very much.

Peter

Thank you, Geoff.

‍