In the final part of my upcoming blog posts titled "Lies, damned lies, and nutrition studies," I've developed some rules of thumb for spotting crappy nutrition studies. Let's see how this one stacks up.

• Follow the money: The NIH; generally impartial; (green)
• Consider the source: The Annals of Internal Medicine is a top-tier journal; (green)
• Size matters: out of the original 148 people who started the trial, only 119 completed it. Is this enough to be meaningful and not only statistically significant? Does the result warrant a larger clinical trial? I honestly don't know how to answer this question. (Yellow)
• Time in a bottle: A 1-year is a long time for a nutrition study, although I’d like to know what the run-in period was. (green)
• If at first you succeed…: The study results need to replicated to ensure that these results weren't a fluke. Yes, I've visited Dr. Eenfeldt's page of studies, but most of them were small and short term. (Yellow)
• The game’s rigged: Was the study designed to produce a desired outcome? In this case, the high-fat arm was unknowingly designed to fail, telling people to primarily unsaturated fats. But fish and olive oil aren’t primarily unsaturated fats. Fish are a low-fat protein source, olive oil is mostly monosaturated, and nuts, as Mark Sisson points out, are not just bags of linolenic acid. Normally, I'd give the study design a red flag. That the study achieved the results it did despite this is amazing, so I have to award the green flag. (green)
• “Knock knock…” “…who’s there?”: The study abstract says that the participants didn't have clinical cardiovascular disease and diabetes. The qualifier "clinical" is a weasel word, allowing for the possibility of sub-clinical presentations of CVD and of metabolic syndrome. I want to know a lot more about the health screening protocol. (Yellow)
• Who’s in Control?: There was no control group — at least I didn't see one mentioned. This is a big no-no in serious research. (red)
• I’m not blind: How do you account for the inability to blind the subject groups and researchers? (Yellow)
• Torture the data: I can’t tell from the lay reporting whether the data analysis techniques, other than the weight loss, fat loss, and basic lipid profile, were appropriate. As far as LDL particle size is concerned, it’s not a useful indicator of anything. Also, there’s no standard way to measure LDL particle size. Frankly, Masterjohn has more credibility on this matter than everyone else I've read put together. (LDL-P is a questionable diagnostic tool, too, but that's a whole other discussion.)(Yellow)
• Believing is seeing:Is there evidence of confirmation bias? Dr. Katz, who foamed at the mouth about this study, has a dog in the fight, as pointed out earlier. But what about the researchers and the academic institutions they represent? I can’t tell from the reporting or the study abstract; there were no statements of conflicts of interest. (Yellow)

There's also Compound Interest’s “A Rough Guide to Spotting Bad Science” to use for further rules of thumb.

So, there are 4 green signals, 6 yellows, and 1 red. The lack of a control group is the most troubling to me. So, I'm not ready to break out the champagne just yet, but I'll raise a toast with a coconut-milk smoothie.