Behavioral studies in the US: an Aversion to Null Findings?

18449Has everyone already read this article that came out in PNAS last month? US studies may overestimate effect sizes in softer research by Daniele Fanelli and John Ionaddis.

Fanelli & Ionaddis gathered a bunch of articles (1,174) that had been analyzed in meta-analyses. Each article fell into the category of Genetics & Heredity or Psychiatry. These categories were stand-ins for non-behavioral vs. behavioral research, and the authors compared the effect sizes reported in individual articles to the overall effect sizes estimated in the meta-analyses.

There were two major findings. First, studies that analyzed behavioral outcomes were more likely to report extreme effects than those that analyzed genetic outcomes. Second, if the corresponding author came from the US, articles were more likely to report findings consistent with their original hypothesis in behavioral but not genetic studies. In other words, they were more likely to find what they wanted to find.

Their interpretation is that the crazy level of publish-or-perish present in the US promotes this kind of thing; preferentially reporting results that fit your hypotheses may be easier in behavioral research, where there is greater variety in methods, replication is harder, and noise tends to be greater. According to the authors, all of these factors give behavioral researchers more “degrees of freedom” to find the results they were expecting.

As anyone who has ever worked in genetics can tell you, the tendency towards publishing only positive findings definitely isn’t limited to behavioral research. It has been reported that genetic studies are subject to the same bias, especially in the US, although Fanelli and Ionaddis didn’t replicate this finding in this (larger) study. And apparently the tendency to publish only positive results has been getting worse over time, presumably increasing as pressure has been mounting in academia.

So what is the answer? Publishing negative results is important, but it’s also a thankless job. First, it’s hard to find a good home for null results. Now that there are journals like PLoS ONE that will publish any scientifically-sound study, regardless of its excitement level, things are getting a little easier… but it will probably never be the case that a null results study is going to be your ticket to tenure or fame. Because of this, even if a null study is well-designed and a scientist knows he/she can publish it somewhere eventually, it’s natural to allocate scarce resources to more exciting/higher payoff studies. This is bad for science, but it’s a natural consequence of the way we’ve set things up.

Fanelli & Ionaddis suggest that this bias may become more common in the rest of the world if other countries  follow our model, which is a scary prospect. We need to figure this out!

Historical Disease Trivia: The Milk Sickness

snakeroot75I’m reading Team of Rivals right now, a book about Lincoln’s presidency. In the section on Lincoln’s childhood, it mentioned his mother died of the “milk sickness,” which was apparently pretty common at that time.

What? The milk sickness? Was this some mysterious, old infectious disease I’d never heard of? I was hooked!

It turns out that no, it is not an infectious disease–although at the time, there was a lot of confusion about that. It occurs when someone is unlucky enough to drink dairy products or eat meat from cattle that have been eating white snakeroot, a plant that contains the poison tremetol. It causes trembling, vomiting, and severe nausea.

Apparently, when settlers from the Northeast started expanding to the south and west in the early 19th century, they weren’t familiar with a lot of the vegetation. They would let their cattle roam around the woods by their homes, not knowing that the white snakeroot they were eating was dangerous. Wikipedia says that this sickness claimed thousands of lives during this period, although I’m not sure how accurate that number is. There were reports that in some settlements, as many as half the people had died of this illness. In the epidemic that claimed Nancy Hanks Lincoln’s life, her great-aunt,  great-uncle, and two neighbors also died.

The story of how the cause of milk sickness was discovered is pretty interesting in its own right. As early as 1811, people had hypothesized that milk sickness was caused by tainted milk, and that the origin of the poison was probably something that cows had eaten.

Apparently the mystery of milk sickness was solved in southern Illinois in the early 1830s, when a female doctor named Anna Pierce Hobbs learned from an elderly Shawnee acquaintance that white snakeroot was responsible. Hobbs had become a doctor to minister to the needs of the pioneers she had grown up among, and at the time, she was the only health practitioner in southeastern Illinois. Personal tragedy sparked her interest in the disease. Milk sickness killed her mother and sister-in-law and almost took her father. After she learned that the cause was tainted milk, she pleaded with her neighbors to abstain from eating and drinking dairy products, and she initiated a successful snakeroot eradication campaign in the region.

However, despite her efforts, when she died in 1869, the medical establishment was still arguing about whether white snakeroot was the cause.  There was a prominent contingent of researchers who were convinced that the illness was caused by some sort of microbe. Reports from the frontlines by people like Hobbs and Shawnee medicine women were discounted. Finally, in 1928, chemists were able to isolate the compounds responsible for milk sickness from tainted milk and snakeroot, and the case was closed.

This is probably a good lesson for researchers today about not being able to let go of pet hypotheses and being willing to seriously consider research coming from unusual sources.

Another Sexual Harassment Scandal in Science

I thought this blog post by Tabitha Powledge was a great rundown of the scandal currently unfolding around Bora Zivkovic, the man who has helped build the vibrant blogging community over at Scientific American.

Basically, it has emerged that a number of women who work with Zivkovic, who is famous as a mentor for science writers, have been subjected to talk and behavior that is majorly inappropriate. Obviously, when this kind of harassment comes from someone who can make or break your career, it becomes a big problem.

Many scientists and science writers were horrified by the initial victim-doubting that met blogger Monica Byrne’s claims (which were quickly substantiated by both Zivkovic and other women who reported even more egregious behavior on the part of Zivkovic).

My guess is that virtually every woman in the field of science or science writing has a similar story to tell–not about Zivkovic, but about being stuck in similar situations, having to deal with similarly inappropriate behavior. It’s a pretty depressing state of affairs. How do we get things to change?

The Truth about Genetically Modified Food?

US-VOTE-CALIFORNIA-AGRICULTURE-FOOD-GMOI know genetically modified food is a controversial topic, but it’s not one I follow closely. Based on what I’ve read, it seems like the genetically modified plants in use are safe–so this isn’t something I worry much about. Recently, I was leafing through the September issue of Scientific American and came across  an article on GM foods that I thought summed up the scientific consensus pretty nicely: The Truth about Genetically Modified Food by David H Freedman.

In this article, Freedman drew a parallel between anti-GM groups and anti-vaccination groups that I thought was both interesting and apt.  How are these groups similar? First, there is the cherrypicking of a few scary studies from mountains of better-designed studies that support GM safety. Second, there is the calling for an impossible standard of evidence for safety as well. Third, there is the tenor of the debate, which often devolves into personal attacks, name calling, and conspiracy theories. Like the anti-vaccination movement, this is a phenomenon that really illustrates how communication between scientists and the public breaks down in a big way. It really makes you wonder what can be done to bridge this gap!

In the same issue of the magazine, there was another article on GM foods, but I totally disagreed with this one.  It was called Labels for GMO Foods are a Bad Idea, and it addressed the growing controversy about whether or not GM foods should be labeled. The argument here was that even though GM foods appear to be safe, if consumers realize that a product contains GM ingredients, they won’t buy that product. Because GM foods are easier to grow, if producers start avoiding them as money-losers, food will get more expensive, which hurts everyone. So don’t provide consumers with labels, and they can’t unfairly penalize GM foods.

This rubs me the wrong way. Denying consumers information because they might use it to make irrational decisions seems pretty Big Brother-ish. Don’t consumers have the right to make irrational decisions about what they eat? f we’re concerned that people’s fear of GM foods is unwarranted, wouldn’t it make sense to give the public the labels they want and simultaneously provide better education about GM safety?

I’d be curious to hear what other people think!