A Data Science Central Community
My print column this week examines the concept of statistical significance — a concept that the Supreme Court recently weighed in on, but that remains elusive even to some scientists who use it to determine whether their experimental findings are worth reporting.
“The concept is so difficult to understand that misunderstandings are forgivable,” said Donald Berry, a biostatistician at the University of Texas M.D. Anderson Cancer Center.
I asked several statisticians to offer definitions of statistical significance. Shane Reese had the briefest one, tailored for a clinical trial for a drug: “It is unlikely that chance alone could have produced the improvement shown in our clinical trial. Because it seems unlikely that chance produced the improvements, we logically conclude that the improvement is due to the drug.” Reese and other statisticians noted that this definition is backwards: It is based on assuming there is no link, then finding the probability that chance alone could have produced the experimental results seen.
Reese and Brad Carlin, who also offered a definition, suggest that Bayesian statistics are a better alternative, because they tackle the probability that the hypothesis is true head-on, and incorporate prior knowledge about the variables involved.
There are other problems with statistical significance. It can be ill-suited to cases where it is unclear if all data is being collected, such as with the reporting of adverse events experienced by users of a drug that is past the clinical-trial stage — or never had to go through clinical trials — and is now on the market. In such a situation, “you have to make a lot of assumptions in order to do any statistical test, and all of those are questionable,” said Susan Ellenberg, a biostatistician at the University of Pennsylvania’s medical school.
“Every statistical test relies on half a dozen assumptions,” echoed Aris Spanos, an economist at Virginia Tech. “Before you use that test, you have to check your assumptions.”
Spanos wishes the Supreme Court had gone further in its recent ruling, in which it determined that a lack of statistical significance didn’t always provide drug companies with enough cover to avoid disclosing reports of adverse events from users of their drugs. Spanos would have liked to see more guidance for how to proceed without relying strictly on statistical significance. “It was a move in the right direction but then you open the system to different kinds of abuses,” Spanos said.Read full story and other similar articles at http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-closeup...