Range Widely
Subscribe
Cover photo

Here’s Why Coverage of the Cash-Improves-Baby-Brains Study is Misleading

If you know what to look for, the tantalizing new report doesn't look as great

David Epstein

Jan 31
17

Last week, I wrote about how everything in your fridge causes and prevents cancer, and how health research should be interpreted with caution. The bottom line was that when researchers have many opportunities to slice and dice a large dataset, it becomes easy to generate spurious correlations; the most interesting spurious correlations then tend to end up in scientific papers and the subsequent press releases.

This week, I’d like to share some pretty serious cautionary notes about an absolutely tantalizing new study that made major headlines — and that I very much hope is completely true! Let’s begin…

The Study

How about I just share the first paragraph of the New York Times coverage of the new study:

Well! I think we can all see why this is a big deal. The researchers don’t yet claim to know why they found these results — whether mothers were able to get better food, or healthcare, or spend more time with their kids, or something entirely different — but scientists quoted in the Times article called the work groundbreaking, and the first of its kind in showing “that money, in and of itself, has a causal impact on brain development.”

Here’s how the study worked: mother/newborn pairs were randomized to receive either $333 a month or $20 a month. The families had an average income below $20,000, so the larger payment was a substantial boost, and could be spent however the mothers wanted. When the babies turned one, researchers administered electroencephalograms, which measure electrical activity in the brain. They reported that the babies of mothers who got the higher payment had more of a certain type of brain activity.

Reasons to Be Cautious

Let me first say that I find this result exciting, and if it is what it purports to be, and it holds up (the study will run at least until the children are four), I certainly think it should have major policy implications. But I want to highlight two reasons to be cautious. The first, short and straightforward. The second, more worrisome, and requiring more explanation.

First: electrical activity in the brain is a proxy for what the scientists actually care about — brain development and cognitive abilities that lead to better life outcomes. The researchers saw more of a certain kind of high-frequency electrical activity in the brains of one-year-olds in the high-payment group. The question, then, is whether that electrical activity is a good proxy of later development. As a Vox article covering the study noted, this type of brain activity has been correlated to language and memory abilities in older kids in other studies. But we really don’t know what this means for one-year-olds. I think the major news coverage did a reasonable job of pointing out that electrical activity in the brain, in and of itself, isn’t the dependent variable we really care about. Now, on to the big one…

Second: Wharton professor Joe Simmons (among others) made a point that entirely escaped the news coverage I read, perhaps because it’s more esoteric. Or perhaps because it might have curtailed the news coverage altogether:

What is Prof. Simmons Talking About?

Let’s zoom in on his point about “pre-registered effects,” because this dovetails with last week’s post. Bear with me for a few paragraphs, because I need to explain pre-registration — and it’s an important issue for thinking about scientific research in general.

Last week, I wrote about how scientists can alight on false-positive findings just by chance when there are many “researcher degrees of freedom” — or ways in which the scientists can group and analyze their data. Since there are many false positive correlations waiting to be found in any large data set, having freedom to slice and dice the data at your whim helps ensure you’ll find some of those false positives.

“Pre-registration” is an increasingly common tactic that scientists use to reduce the amount of freedom they have. (“Pre-registration: When you want less freedom.” Not going to win an election campaign, is it?) Specifically, when scientists pre-register a hypothesis, it means that before the study begins they specify what effects they will look for when they do their data analysis later on. In other words, they can’t just go plunging through their data looking for any old positive correlation and claiming to have predicted it, because they told their peers ahead of time what they would look for.

Last week, I used the hypothetical example of a football commentator who says that the Chicago Bears are undefeated after a bye week when they wear their alternate jerseys. When you look at the past, sifting data for stats like that is easy; they abound just by chance. If the commentator instead predicted that the Bears would henceforth go undefeated after bye weeks while wearing alternate jerseys, and the commentator was right, then that commentator might really be on to something. It could still be luck, but it’s more likely the commentator uncovered some real cause and effect.

Intuitively, you probably get this, in a way. You’d probably be more amazed if a commentator predicted an effect and turned out to be right than if the commentator told you about the past and assured you it could predict the future.

Turns out, when clinical trials have pre-registered hypotheses, they tend to produce a lot fewer positive results. Take a look at the figure below, from a 2015 study of drug and supplement trials for cardiovascular disease. Each circle on the chart is a single drug or supplement trial, and the x-axis shows the year the trial was done. The circles with a plus sign are trials that found a beneficial effect of the drug or supplement; the solid blue circles are trials that found no effect.

Drugs and supplements seemed to work a lot better prior to 2000! Y2K drug trial bug?? No. That’s the year that the National Heart, Lung, and Blood Institute began insisting that researchers it funded pre-register the effect they were planning to study. Again, that meant that the scientists had to specify before the study if, say, their plan was to see whether a drug prevented heart attacks in women over the age of 60; they couldn’t just collect data and then retrospectively look for positive correlations.

As you can see, when researchers have to specify ahead of time what they’re looking for, they’re a lot less likely to report back with positive results.

Getting back to Prof. Simmons’s tweet, he’s saying that the largest effect of payments on brain activity in this new study was not pre-registered, and thus we should treat it with caution. Furthermore, he’s also saying that the effects that were pre-registered were found not to be statistically significant. Raise the caution flag. Effects that the scientists specified in advance were not found, while the most headline-worthy one was not specified in advance.

In fact, looking at the actual study results — below, showing different types of brainwaves — once appropriate statistical corrections were made, none of the results meet the traditional standard to be considered “statistically significant.” None of them. “The results are interesting, but none were statistically significant” is a far cry from the national headlines that are now collected on the study’s official site.

The traditional standard for statistical significance — a measure of how likely a result would have been to occur by random chance — is a P value of less than 0.05. None of the results met that threshold. The one that was closest (absolute beta) was not preregistered.

This does not mean that the results are certainly false. And, as Simmons notes in the very next tweet, “I do think that giving (enough) cash to poor mothers is very likely to have beneficial behavioral and developmental effects."

What Does This Mean For The Study?

Overall, the “Baby’s First Years” study is a momentous and important effort. I agree with Simmons; I think cash payments above a certain threshold are likely to have a host of beneficial effects on children, whether or not those have to do with infant brainwaves.

I’m also eager to follow the study. But the results published so far are not, in fact, strong evidence of the headlines, or of scientists’ claims in news articles.

The good news is that the scientists went out of their way to be transparent. They openly shared their data, and the process of their analysis, in addition to pre-registering their hypotheses. (For more info on their brainwave-analysis pre-registration, see section SI4 here. And to see other effects they'll examine, see this.) The reason that it's easy to critique their work quickly is because they openly shared how it was done, allowing others to work with the data. Kudos to them! The bad news is that the news coverage, including the scientists’ own quotes, will leave readers with an unfairly definitive conclusion.

Thank you for reading. If you found this post useful, and would like me to keep writing on science news from time to time, please share it. (If you’re reading in email, here’s a link you can share.) I’ve realized that reader shares are really the only way that this newsletter spreads. And if a friend sent this to you, you can subscribe here. Until next week...

David

Subscribe for free to Range Widely
By subscribing, you agree to share your email address with David Epstein to receive their original content, including promotions. Unsubscribe at any time. Meta will also use your information subject to the Bulletin Terms and Policies
17

More from Range Widely
See all

Can Bad Number Sense Amplify Deadly Hate?

Here's a dire reason to be curious about the numbers around you
May 18
4
6

The Nazis Invented Marriage Counseling, And Other Surprising Relationship Research

Q&A with the bestselling author of a new book on the science of relationships
May 9
4
19

Here Are Two Tips for Picking Out Misleading Stats in the News

I used these, and it led to this newsletter’s first official correction of a scientific journal article
May 2
7
13
Comments
Log in with Facebook to comment

17 Comments

  • Ian Bremmer
    Writes GZERO World with Ian Bremmer
    Great breakdown, David.
    • 14w
  • Jenna Ross
    I used to live in the UK and got a cash “child allowance” for each child monthly. I wasn’t at a point in life to dive into policy, but was told the reason it went to the Mums is because then it went to the kids. If it went to the fathers it would all b…
    See more
    • 15w
    • Edited
    • Author
      David Epstein
      Thanks for that, Jenna. My guess (and hope) is indeed that such cash payments (above some threshold) will have beneficial effects, but I'm nonetheless concerned about the way this portion of study has been disseminated and digested. Given the profound …
      See more
      • 15w
  • John Lockney
    The short version:
    Like I've been saying for years, the spoon-bending machine…
    See more
    • 15w
    • Author
      David Epstein
      John, appreciate this comment. I agree about fMRI. Of course, every area of science (or inquiry in general) is subject to misinterpretation and over-extrapolation, but fMRI seems to be a class apart in recent years. I wonder to what degree the idea of …
      See more
      • 15w
  • Monica C.
    I’m a scientist and you are gifted in explaining these concepts. I read the study as well but your analysis really put the conclusions in perspective. And massive kudos on your warmth and kindness towards these scientists. It’s really difficult to cond…
    See more
    • 15w
    • Author
      David Epstein
      Monica, that is very kind of you to say, and especially gratifying to hear from a scientist! I really do think the "Baby's First Years" study is a tremendous (and tremendously important) effort. And I've read work by some of these researchers before, a…
      See more
      • 15w
  • Bekah Lin
    As a research nerd, my thoughts are this: do we have data on the health of the pregnancies? Do we have data on the health of the babies at birth? Stats on prematurity? Stats on drug abuse status of the families included? Do we know what families spent …
    See more
    • 15w
    • Author
      David Epstein
      Hi Bekah, I appreciate research nerdery! First, I'd say that the correlation here is at question in the first place, before even causation. But I will say this is a monumental research effort, and many characteristics of the families included are well …
      See more
      • 15w
  • Matt Thomas
    Thanks for this post, David! When running your own studies before you switched into journalism, did you ever encounter this problem? How did it force you to refine your thinking/practice as a scientist ahead of time?
    • 14w
    • Author
      David Epstein
      Hey Matt! Back when I was working in science, I really wasn't aware of these issues. As crazy as this sounds, it just wasn't part of the education of most scientists-in-training. (Unfortunately, I think it still isn't, even while there's much greater a…
      See more
      • 14w
  • William Murphy
    Hi David, many thanks for this. I love this topic.. that is really interesting how the pre-registration is working! I imagine this can only be a good thing if it is leading to less false positives and less results caused by "researcher degrees of freed…
    See more
    • 15w
    • Author
      David Epstein
      Hey William, I absolutely do think this is part of a reaction to the replication crisis. I think, though, an increase in pre-registration predates the period where "replication crisis" started being used widely. In part because it became more common fo…
      See more
      • 15w
      • Edited
    View 1 more reply
  • Silvana Ordoñez
    Putting this in my critical thinking box- thanks you. If you had gotten a press release of the study, and your editor had asked you to write a piece about it. Would you have written it? if so, curious to know what the headline you'd write if this was your assignment.
    • 14w
    • Author
      David Epstein
      That's a great question. For starters, I definitely would have read the actual paper, and given my science background, I think the data analysis would have raised my eyebrow. That said, earlier in my science writing career, I was not aware of the pre-r…
      See more
      • 14w
      • Edited
    View 1 more reply
Share quoteSelect how you’d like to share below
Share on Facebook
Share to Twitter
Send in Whatsapp
Share on Linkedin
Privacy  ·  Terms  ·  Cookies  ·  © Meta 2022
Discover fresh voices. Tune into new conversations. Browse all publications