A British Medical Journal study and companion editorial take aim — at exaggerations and misstatements of medical fact — from investigators, academic institutions, and the press

© 2014 Peter Free

Citation — to study

Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, Fred Boy, and Christopher D Chambers, The association between exaggeration in health related science news and academic press releases: retrospective observational study, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7015 (10 December 2014)

Citation — to editorial

Ben Goldacre, Preventing bad reporting on health research, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7465 (10 December 2014)

One of my peeves is now apparently shared by the British Medical Journal

Researchers, academic institutions and the press all frequently mislead the public with exaggerations and misstatements of fact and causation:

Although it is common to blame media outlets and their journalists for news perceived as exaggerated, sensationalised, or alarmist, our principle findings were that most of the inflation detected in our study did not occur de novo in the media but was already present in the text of the press releases produced by academics and their establishments.

Among biomedical and health related press releases issued by Russell Group universities [see here] in 2011, 33% to 40% contained exaggerated statements compared with the corresponding peer reviewed journal articles.

Moreover, when press releases contained exaggeration it was likely that the news would too . . . 58% for advice, 81% for causal claims, and 86% for inference to humans . . .

but

when press releases did not contain exaggeration, rates of exaggeration in news were only 17%, 18%, and 10%, respectively.

Therefore the odds of exaggerated news were substantially higher when the press releases issued by the academic institutions were exaggerated . . . odds ratios 6.5 [medical advice], 20 [causal claims], and 56 [inference to humans], respectively. . . .

Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, Fred Boy, and Christopher D Chambers, The association between exaggeration in health related science news and academic press releases: retrospective observational study, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7015 (10 December 2014) (at 1st paragraph under Discussion (paragraph split, underlines added)

How did the research team discover this?

Ben Goldacre explained that:

Sumner and colleagues identified all 462 press releases on health research from 20 leading UK universities over one year. They traced 668 associated news stories and the original academic papers . . . .

[T]hey assessed the press releases and the news articles for exaggeration, defined as claims going beyond those in the peer reviewed paper.

[T]he authors’ structured appraisal focused on three areas: making causal claims from correlational findings in observational data, making inference about humans from studies on other animals, and giving direct advice to readers about behaviour change.

If a news story claimed a new treatment for humans, for example, but the study was on mice—and the academic paper made no claim about humans—then did the exaggeration first appear in the press release, or the newspaper article?

© 2014 Ben Goldacre, Preventing bad reporting on health research, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7465 (10 December 2014) (extracts)

The academic institution’s initial presentation apparently makes a big difference downstream

Presumably because most reporters are not scientifically confident — or perhaps because some do not think at all:

For main news statements about correlational results, 39% . . . were more strongly deterministic than those present in the associated journal article. The odds of exaggerated statements in news was 20 times higher . . . when press release statements were exaggerated . . . than when they were not . . . .

For non-human studies, 47% of news contained inflated inference to humans. The odds of exaggeration in news was 56 times higher . . . when press release statements were exaggerated . . . than when they were not . . . .

© 2014 Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, Fred Boy, and Christopher D Chambers, The association between exaggeration in health related science news and academic press releases: retrospective observational study, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7015 (10 December 2014) (at last 2 paragraphs under section entitled, “Association of news exaggeration with press release exaggeration”) (extracts)

Team’s reasonable assumptions about the nature of exaggeration

Understandably:

[W]e assume that pressure to publish means that most journal articles already contain the highest level (at least) of justifiable inference and advice; if further inflation occurs in a press release, it is thus likely to go beyond what a consensus of scientific opinion would find acceptable.

Consistent with this interpretation, a preliminary survey (see supplementary section SI12 and figure S2) revealed that a surprising number of scientists were willing to say that their press releases were exaggerated (relative to their own judgment of what was scientifically justified).

Furthermore, given the imperfections of peer review, many journal articles may contain statements that are already exaggerated relative to a consensus of scientific opinion, or at least spun to emphasise positive findings,and thus our measured level of within university exaggeration is likely to underestimate the extent of the problem.

© 2014 Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, Fred Boy, and Christopher D Chambers, The association between exaggeration in health related science news and academic press releases: retrospective observational study, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7015 (10 December 2014) (at last 2 paragraphs under section entitled, “Association of news exaggeration with press release exaggeration”) (extracts)

“Yeah, I lied and I’m darn proud of it!”

In writing BrainiYak, it is rare for me to come across a scientific paper that appears to underemphasize the nature of its findings. Publication pressure, narcissistic self-promotion, and the weight of anticipated profit all run the other way:

It is important that these results are not perceived as simply shifting the blame from one group of non-scientists (journalists) to another (press officers).

Most press releases issued by universities are drafted in dialogue between scientists and press officers and are not released without the approval of scientists (and confirmed in our survey, see supplementary section SI12), and thus most of the responsibility for exaggeration must lie with the scientific authors.

The blame—if it can be meaningfully apportioned—lies mainly with the increasing culture of university competition and self promotion, interacting with the increasing pressures on journalists to do more with less time.

© 2014 Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, Fred Boy, and Christopher D Chambers, The association between exaggeration in health related science news and academic press releases: retrospective observational study, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7015 (10 December 2014) (at 1st paragraph under section entitled, “Implications for practice”) (extracts)

The laughable lapse of scientific integrity in original investigators that the research team recorded in its Supplement

To wit, at S112:

Scientist doublethink

Whilst instigating the main study, we performed an online survey of scientists’ attitudes toward science in the media, and their experiences with PR [press releases]. We advertised the survey via the Guardian, the BBSRC, and social media.

The sample is self-selected and likely biased towards pre-existing interest in the topic of science news and by the subject area distribution in our advertising routes. As expected, the respondents (N=248) blamed journalists more than any other party for misreporting in science news.

However, 79% of scientists who had PRs about their work reported involvement with those PRs, and despite this involvement, 32% acknowledged that their PRs were exaggerated (Figure S2).

Thus it appears that some scientists do have awareness that PRs are a source of misreporting, but as a group we appear to engage in doublethink – colluding in producing exaggerated PRs but mainly blaming the media for the shortcomings of science news.

© 2014 Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, Fred Boy, and Christopher D Chambers, The association between exaggeration in health related science news and academic press releases: retrospective observational study, British Medical Journal [BMJ] 349:g7465, DOI: 10.1136/bmj.g7015 (10 December 2014) (at Data Supplement link to PDF section entitled, “Supplementary Information, S112.Scientist doublethink”) (paragraph split, underlines added)

The moral? — Modernity’s twin soul diseases of Ratbag Capitalism and Narcissism roll confidently along

It was inevitable that science and medicine become victims. We are professionally no better than who we are.