Torturing the Data: How to Make Sure Your Scientific Research is Significant

The scientific method looks a whole lot different in reality than in theory

I loved science when I was a kid—especially biology. I loved learning more about nature, going out in the woods to look at plants and insects, attending Science Camp every summer at Michigan State University’s Kellogg Biological Station. When my high school biology book came in the mail, I read through the whole thing before school even started. I took seven science classes in high school and tested out of three semesters of college biology and chemistry.

With all that studying, I thought I knew what science was. It was an objective search for truth. That’s what all the books I read as a kid and teenager said. They all gave the same outline of the scientific method: scientists observe nature, make a hypothesis based on their observations, do an experiment to test that hypothesis, analyze the data to see if it’s significant, and then report whether or not the results confirm the hypothesis. If the experimental results don’t match the hypothesis, they revise the hypothesis to better fit the data and start the whole process over again.

My faith in the objectivity of science remained relatively intact through my first two years of college. I carefully took notes on lectures so I could put the right answer on the test. When some of our lab experiments didn’t turn out as expected, I just assumed it was because we were inexperienced and messed something up. We were just undergrads, after all; I figured the grad students and professors were much more careful and got much better results.

Then I took a graduate-level statistics class. The professor taught us a bewildering array of statistical methods—analysis of variance, least significant difference, randomized complete block designs, and more. The math was complicated, but I was good at algebra and didn’t have too much trouble understanding it. What surprised me was when the professor told us that we should choose whatever statistical analysis was most likely to make our experiment have significant results.

“If you torture the data enough, it will confess.” That was his key takeaway from the class. He repeated it every week. Results from your first analysis not significant? No worries; just keep trying different analyses until something significant showed up. Basic analysis not significant enough? Try something more complicated, like principal component analysis. Just feed your data into statistics software, and it will spit out colorful, complicated graphics ready for publication in a scientific journal. We spent far more time learning how to use R, an open-source statistics program, than learning the theory behind the equations or how to derive them.

Why all the stress on significance? Simple: Only studies with significant results get published in scientific journals. The rule of academia is, “Publish or perish.” Whether the significance actually reflects something real or is just an artifact of analysis is irrelevant. The important thing is that it’s significant. That’s enough to get it published.

It was the first intimation that maybe what I’d learned in school about the scientific method didn’t accurately reflect what real scientists were doing at real research institutions. “If you torture the data enough, it will confess” didn’t sound at all like my rosy vision of disinterested objectivity. But I still thought that maybe I just didn’t know enough yet. I was still just an undergrad, even though I was a senior. So I applied to graduate school, excited that I would finally be getting into the inner sanctum of a research laboratory and doing “real science.”