Range Widely
Subscribe
Cover photo

Here's a Tip for Picking Out Misleading Science News (Featuring AC/DC and the Beatles)

Another sniff test for detecting bad science and bad science reporting

David Epstein

Apr 18
6
44

Bad science, or bad reporting on science is a recurring topic here at Range Widely.

Today I want to highlight an example of really bad reporting on top of what looks like quite bad science.

“Surgeons who listen to AC/DC are faster and more accurate,” according to a bunch of headlines. But let’s pick on the New York Post. Here’s how the Post put the findings from a surgical journal:

Big if true.

Now, before we get to what the actual study said, I want to hammer home a point I’ve made previously: there are often clues in news articles that a study is not reliable. A typical clue is when a finding proclaimed in the headline is explained later in the story as something that only works in newly discovered, very specific conditions. Back to the Post:

That’s interesting. Apparently volume mattered differently for the Beatles than it did for AC/DC. So I read coverage in another publication, which added this tidbit:

Curiouser and curiouser, as Alice would say. So the surgery-enhancing effect of soft rock only works at low volume, while the effect is most pronounced for hard rock at high volume. Well of course! That’s how the music was meant to be played, and anything else would upset a surgeon’s worldview in the middle of surgery, right? Unlikely.

In January, I wrote a post on the famous “everything in your fridge causes and prevents cancer” study. I won’t go into it in detail here, but the upshot was: I recommend that when you see this sort of ultra-nuanced effect in a news article, it may be a sign that the researchers inappropriately (but often not maliciously) sliced and diced their data in order to create some tantalizing positive finding, which — given enough data and enough slicing and dicing — they will inevitably find among the many possible false positives.

Let’s say a study starts out asking whether music makes surgeons perform better, but the results show nothing. Dead boring. So then the researchers separate the data into surgeons who heard soft rock versus hard rock. Ok, now the hard rock shows an effect. But, hmmmm, still no effect for soft rock. But if you look only at the data when the music is low-volume, there’s an effect. Interesting! Headlines!

But the reality is often that the researchers just sifted the data so much that they were bound to find false positives. Again, I detail this process at greater length in this post. Whenever you get a whiff of slicing and dicing, raise your eyebrow.

But there are other problems in this particular case…

There Were Neither Surgeons Nor Wounds

The New York Post (and others) wrote that the surgeons got better at “stitching up wounds.”

According to the actual study, there were no surgeons, and there was no wound stitching. Rather, 30 medical students in Dresden “who showed a particular interest in surgery” were tested on four different tasks. Like “peg transfer”:

And “precision cutting”:

I’m sure those are fine tasks, but they’re not “stitching up wounds.” But aside from the possibility of data sifting, and the fact that the not-surgeons were doing not-surgery, I want to cast a bigger aspersion on this study….

The Practice Effect

Some famous psychology findings have esoteric names. Not the “practice effect.” It means that you will often improve at a new task with practice. That might seem trivial, but it’s important for psychologists to recognize.

Let’s say I’m doing a study about whether eating bananas improves the skills of student drivers. I have students drive around, and I rate their driving. Then I give all of them bananas, and have them do the driving test again. They do better this time. Before jumping to the conclusion that bananas lead to better driving, I’m definitely going to want to know how much people normally improve upon taking the driving test repeatedly.

So here’s a figure from the surgery study:

The med students were slowest at cutting tasks (that’s “complex II”) with no music, medium-fast with medium-volume hard rock, and fastest with high-volume hard-rock. But, according to the study:

That’s a big deal. It means that every med student did the music trials in the same order. With the practice effect in mind, would you like to guess what that order was? Take a minute…

…If you guessed that first they had no music, then they had medium-volume music, and finally high-volume music, you win. (I think. The paper is a bit confusing on this point.) I have helpfully annotated the figure for what probably happened:

As Allen Iverson and Ted Lasso would say: We’re talking about practice.

Thank you for reading. If you think this post might be good for science-news literacy, please share it. Here's a link. (Tag me on Twitter or Instagram so I can thank you.)

And thanks to Jeff Novich for sending me the New York Post article, and to Drew Bailey for always being up for a chat about research methods.

As always, you can subscribe here.

Until next week…

David

P.S. Having read the paper again, it is — to use scientific terminology — a hot mess. Let me know in the comments if you’re interested in another (shorter) post about how the main finding touted in these music/surgery news articles isn’t real, according to the scientific paper itself. In that hypothetical post, I’d also explain how you can catch the mistake in the paper, even though the authors missed it. And if you made it this far, congrats on your attention span!

Subscribe for free to Range Widely
By subscribing, you agree to share your email address with David Epstein to receive their original content, including promotions. Unsubscribe at any time. Meta will also use your information subject to the Bulletin Terms and Policies
6
44

More from Range Widely
See all

Can Bad Number Sense Amplify Deadly Hate?

Here's a dire reason to be curious about the numbers around you
May 18
4
6

The Nazis Invented Marriage Counseling, And Other Surprising Relationship Research

Q&A with the bestselling author of a new book on the science of relationships
May 9
4
19

Here Are Two Tips for Picking Out Misleading Stats in the News

I used these, and it led to this newsletter’s first official correction of a scientific journal article
May 2
7
13
Comments
Log in with Facebook to comment

44 Comments

  • James Hamblin
    Writes THE BODY
    Phew. When I saw the email subject line I worried you were giving feedback on something I wrote.
    • 4w
    • Author
      David Epstein
      I need more time to finish that 17-part series
      • 4w
  • Matt Thomas
    Thanks for this one, David! I always appreciate your posts about increasing scientific literacy. I'd love to see that additional post you mentioned in the P.S. On the subject of experiments, I know you've been doing this newsletter for a little while n…
    See more
    • 4w
    • Author
      David Epstein
      Hey Matt, thanks! I'm going to do that second post soon. As far as advice, I've gotten loads of good advice, although I must say, I haven't followed much of it. For example, to focus a newsletter so that I can describe what it's about in a sentence. I …
      See more
      • 3w
    View 3 more replies
  • Tina Flint Huffman
    Correlation does not imply causation - research 1A.
    2
    • 4w
    Eric Janik replied
      ·
    1 Reply
  • Sean D'Souza
    In some ads, you’ll see the fine print: tested on 8 people. I bet this has a testing ratio that’s similar. And they left out country music and Bollywood. 😉
    • 4w
    Sean D'Souza replied
      ·
    4 Replies
  • Annah Kuriakose
    Hey David! I teach a small high school leadership group, and we're about to discuss mental health x digital wellness, including taking a critical eye to information we see online. Interested in coming to chat with them (virtually)? Let me know! Thanks for your work!
    • 4w
    • Author
      David Epstein
      Hi Annah, that is so cool! I'm really glad to hear that, but have been getting quite a lot of requests to Skype into small groups lately, so had to put that on pause to try to claw back from being behind on my own work. I hope you know I truly apprecia…
      See more
      • 3w
    View 1 more reply
  • Stacey Morris
    Fantastic! Since the vast majority of Americans get their info on scientific studies from mass media (never the actual papers), it's brilliant to share how to spot the unreliable data in a newspaper article.
    • 4w
    • Author
      David Epstein
      Thanks Stacey! I'm sort of hoping to do posts here and there that highlight strategies like this. I'm not so much concerned with this particular study as trying to get some useful case studies. Glad you appreciated it!
      • 4w
  • Simone Azevedo
    Now, do they mention the age range of those "surgeons"? I was just imagining if I was the surgeon... probably playing disco would make me move faster and better.
    • 4w
    View 1 more reply
    • Author
      David Epstein
      haha...I don't recall seeing an age range, only that they were trainees. I do hope someone realizes the necessity of a disco surgery study, though.
      • 4w
  • Nellie Ide
    Hi David, this is super interesting! I am in a neuroscience class right now where we are analyzing scientific articles and we often spend more time pointing out limitations than discussing the findings. I have been very surprised by the number of flaws…
    See more
    • 4w
    • Author
      David Epstein
      Hi Nellie! I agree with that in general, but in this case I have a hard time even seeing what important question is raised. I guess we could argue the surgery/music question is important, but if merely posing a question and then adding no value whatsoe…
      See more
      • 4w
    View 4 more replies
  • Eric Janik
    If I read it in the New York Post I'm likely not to believe it anyway.
    • 2w
  • Nate Kornell
    This is great stuff, I mentioned it in class today. Always be skeptical, chapter 9 million
    • 4w
    • Author
      David Epstein
      Hey Nate! Thanks for the note...and now I'm going to pretend I don't know you're reading this newsletter so I don't get too self-conscious about research critiques;) But I'm sure you'll tell me politely when I screw one up. Anyway, super cool to hear I…
      See more
      • 4w
View 3 more comments
Share quoteSelect how you’d like to share below
Share on Facebook
Share to Twitter
Send in Whatsapp
Share on Linkedin
Privacy  ·  Terms  ·  Cookies  ·  © Meta 2022
Discover fresh voices. Tune into new conversations. Browse all publications