I've noticed some people getting frustrated with the tendency of the anti-Atkins media (or even the ambivalent-Atkins media) to describe differences or results of pro-Atkins studies as "not significant." I just want to clear something up for all those of you who are not True Science Nerds Those of you who *are* True Science Nerds, feel free to tune this out or correct me. Those of you who aren't, give me a smack if this sounds preachy, I'm trying to be precise:

"Significant", in the context of a scientific study of some kind, is a loaded word. It doesn't mean what it means in normal, everyday conversation. It means that when a statistical "test" -- basically, some kind of formula invented by somebody who likes math a lot more than me -- was performed on the data, there was a very low probability of the results of the study being random and meaningless. You can get that low probability in basically two ways. The first way is if the difference in the study is really big. This would be something along the lines of having 50 low-fat dieters and 50 low-carb dieters, where the low-carb dieters all lost 50 pounds and the low-fat dieters all lost 5 pounds. The second and more common way is if there are lots of small differences. So 5,000 low-carb dieters all lost 20 pounds, and 5,000 low-fat dieters all lost 15.

Now, here's where it gets annoying. An "insignificant" difference would be something along the lines of 5 low-fat dieters losing 15 pounds each, and 5 low-carb dieters losing 20. There's still a difference, it's just not *statistically* significant. So while we all know the truth, the scientists aren't allowed to give as much weight to that study.

Unfortunately, in the translation from a scientific study to the mass media, "insignificant" loses that special meaning and people start thinking it means "not important" or "very small". Of course I'm preaching to the choir here, but I think low-carb works way way better than low-fat, and so I'd bet you at least a dollar that in the case of low-carb studies, most "insignificance" is just due to a lack of participants. Solution? Let's all start signing up for low-carb studies -- and let's make sure the anti-Atkins people know what "significant" really means. After all, most of *their* studies aren't significant, either. Even the ones *with* lots of people.

Thanks for posting that, Bookery. Even for folks who know the basics of "statistical significance", it is frustrating to see journalists use the terms signicant or insignificant in articles.

Kay

When bad science happens many things can get worse. There was an EPA study (after the federal courts looked at it they said that it was biased and flawed because they picked and chose the data they wanted) that showed second hand smoke will kill you, but the results were statistically insignificant. Well, the non-smokers just looked at the results they wanted. So now alot of cities are banning bar and restaurant smoking based on a flawed study. And even though the courts said this was a flawed study it is still cited by people who want to show smoking is bad. And the number of people who die from secondhand smoke keeps getting larger...you just have to get creative with you data gathering. If your roommate smokes and you get hit by a bus...another victim of secondhand smoke. I am just woried that someone is going to start banning any source of smoke ie campfires, barbeque grills just to keep people safe from that evil smoke.

Before you flame me, I know that smoking is not good for you, but secondhand smoke has a very small (if any) effect on others.
If they want to do a better study, they should check the cancer rates of bar and restaurant workers, casino workers, and others that have to work in smoke filled enviornments. And then compare that to the general population.

Just to add one more wrinkle to the discussion of statistical significance: in addition to statistical significance being a function of the sample size and the difference in effect size between the two samples, whether or not a difference is statistically significant is also dependent upon the variance in the data. In other words, a difference of five points between two different groups may be significant in a sample of size X if the data are very tightly clustered -- everyone in one group lost between 4 and 6 pounds while everyone in the other group lost between 9 and 10 pounds -- and may not be significant in a sample of the same size if the data are more spread out -- everyone in one group lost between 1 and 9 pounds while everyone in the other group lost between 6 and 13 pounds. In both cases, the average difference between the two groups is five pounds, but in the case where the data are more spread out, you need a bigger sample to be certain that the difference was not due to random chance.

What statisticians are doing when they are measuring statistical significance is computing the probability that the difference between two groups is greater than 0. A difference is usually considered to be statistically significant if we are 95 percent certain that the difference is greater than 0. The reason we have to do this is that anytime you select a random sample, there is a chance that it is not really representative of the population. However, the more people you select into the sample, the more likely the sample is to be representative. If you think about tossing a coin -- if you toss is three times you may get heads all three times. If you toss is 300 times, it is unlikely you will get heads more than half the time -- it evens out. The same thing happens when you pick a sample -- when you pick a large one it is more likely to be representative and thus it is easier to prove the statistical significance of your results. Unfortunately, larger samples cost more money!

In a good statistical study, statisticians will compute the expected effect size first -- they will figure out how much of a difference between the two groups there is likely to be -- and then make sure that they have a sample big enough to show significance at that difference. Otherwise, there is little point in doing a study. However, unfortunately this is sometimes not done or the effect size (the difference between the two groups and the variance in the data) may be smaller than expected. Therefore, we may not be 95 percent confident the difference is statistically significant -- we may only be 75 or 80 percent confident that the difference is statistically significant and that is usually not good enough for publication. It doesn't mean that the difference isn't real; it just means that we aren't sure whether or not the difference is real.