Research: Contradictory, Misleading, Inscrutable?
Disclaimer: I like research. I think it’s essential, wonderful, and we need more - but like lots of people, I frequently struggle with the fact that it often seems difficult to find research results from which we can draw satisfying, unambiguous, and even (dare I say) useful conclusions. Research findings are often conflicting: they contradict each other or compete with each other. They can be misleading, tempting us to draw all the wrong kinds of conclusions. And they can be impenetrable. Sometimes it seems like academic publications must be bent on discouraging normal, everyday language.
Many people, even those working in academia, find out about new research second hand or through the media. However all too often what we hear about one week doesn’t match with what’s in circulation the next. Let us take the example of an area of research that many of us care deeply about – whether red wine is good or bad for one’s health. The media is very, very confused about whether red wine helps us to live longer or reduces life expectancy; is good for our hearts or puts a strain on the cardiovascular system; causes or protects us from cancer, and more. With so much confusing, conflicting and too often, disappointing research out there, what should we do? It may be tempting to stop believing in the utility of research all together but that would mean missing out on some of the truly great work being done in academia that might impact us personally or be highly applicable in the workplace. Instead, here are three approaches to research hearsay that you might like to use:
1. Apply a critical filter: we should be wary about what we hear second hand or read in the mainstream media when it comes to academic findings, especially when we either like how it sounds or when it seems too good to be true!
Unfortunately accurate science communication is rare on major news sites. Their business is getting people to look at their output rather than accurately reporting science news. This of course increases the likelihood that they will sensationalise stories (of all kinds) in order to gain readership, and reduces the likelihood that they will go out of their way to do more than superficial analyses on new research findings that sound like good clickbait.
Case in point, in 2015 a journal published research that found that chocolate is actually good for us and can help us to lose weight. It made news internationally and was widely reported on. Media outlets everywhere jumped at the chance to share such a click-worthy story without applying any critical lens to the research. As it turns out, this incredibly flimsy piece of research was retracted and, even worse, its terribleness was deliberate!. The authors published and promoted their flawed research as more of a social experiment to demonstrate the shortcomings of science journalism and the journal publication process. Despite how transparent the authors have been about their intentions and the non-credibility of their findings, many of the original media stories lording chocolate as a weight loss tool are still out there (e.g. Daily Star UK, Times of India)!
Further, to avoid the trappings of confirmation bias (the tendency to interpret evidence or events to match our own pre-existing views) we need to be extra cautious if the news neatly corresponds with our own views or wants (oh how much I would like to believe that chocolate will help me lose weight and red wine will let me live forever!). It’s also sensible to think critically about any reports of ‘radical’ findings that seem to turn previous knowledge on its head – in the vast majority of cases, science is built on earlier research and slowly expands in the same direction, there’s not that much room for ‘radical’.
2. Go straight to the source – it’s easier than you think and can be very illuminating
Media reports of research findings should be verifiable by googling the authors name(s) and the subject area, especially using a research specific search engine such as google scholar. Check whether the research paper’s abstract matches the claims being made in the media (in my experience the result is frequently something similar, but drastically toned down and with many ‘ifs’ and ‘buts’). Better yet, read the entire article if it’s available and look for links to similar articles to see if other research is saying the same thing.
Pay special attention to the conclusions and limitations section where the researchers should own up to where their research falls short. Scrutinise the qualifications of the researchers and the institution(s) from which they hale. Importantly, check to see if they report any funding sources that may indicate a conflict of interest.
Still unsure? Sometimes research findings aren’t communicated well, even when going to the source. If it’s important, it’s always an option to email the researchers directly or find their number via their research institution and call them – they’re just people and most of them want to have a real impact on the world.
3. Empower ourselves by finding out what makes good research and apply that knowledge.
Going a step further, we can learn to pick up on the indicators of valid, reliable and thorough research. There are many different types of research, each with pros and cons – strengths and weaknesses. There are also numerous tell-tale signs of ‘good’ and ‘bad’ research (and much that is in between). Read about what makes good research in our next blog in this series.
So, when should we listen to researchers?
The bad news is that ultimately we all need to take some responsibility for what we buy in to and do a bit of ground work to ensure that we, or the organisations we are a part of, do not become a part of the problem. The good news is that we can feel more confident once we have:
Applied some critical thinking and tested our own inclination to confirmation bias
Found the source of the research being reported on and verified that their claims have been conveyed accurately, and that their credentials stand up to scrutiny.
Learnt what makes good evidence and how to identify high quality from low quality research
This may seem like an onerous process, but following these steps for a single piece of research can take mere minutes and avoid embarrassing, or costly, mistakes later. Help is always available when it comes to finding and critiquing a whole body of research on a topic, or when looking into highly complex issues. In the meantime, enjoy chocolate and red wine and hope for the best!