## Do Doctors Understand Statistics?

I was delighted today to see an article in the Washinton Post that follows up on several health stories that were recently published on low-fat diets. I have been struggling trying to understand what is really happening from the reports in the popular news media. There seem to be only two types of publications available, the popular literature that ducks the scientific concepts like control groups and the mathematics required to discuss the statistics, or the full blown specialist article that can only be read by someone working in the field. Today’s article seemed to be aimed somewhere in the middle which gives me and opportunity to understand the issues better. It is written by doctors from the VA Outcomes Group which is associated with Dartmouth Medical School.

In a broad based study with 50,000 women, low-fat diets did not reduce the incidence of breast cancer. A smaller study of women who had already had breast cancer showed a benefit from a low-fat diet. Today’s article discusses the statistical judgments made in each case.

The p values for the effect of low-fat diet on breast cancer in the two studies were quite similar. For women with breast cancer, the p value was 3 percent. For women without breast cancer, the p value was 7 percent.

Now the p value is the probability that the effect could be due to chance, so a smaller value is better for indicating a real effect. There is an accompanying article that discusses statistics and it makes clear that they are talking about Gaussian statistics. They then make a statement that struck me as just wrong.

Since the p values are actually quite close, we would argue that the role of chance was about the same. That is, if you believe one is real, you should probably believe the other is real.

These two values are not nearly the same. The experiment having a p value of 3 is more than twice as likely to be not due to chance than the one with a p value of 7. However, the real problems is that neither of these values is very strong. Physicists require there be much less than 1% chance of the result being due to chance before it is accepted as real. With a p=5 as the cutoff then one experiment in 20 will be wrongly interpreted.

Technorati Tags: Science

March 15, 2006 at 11:44 pm

Just another data point – when I did statistical quality control (Six Sigma stuff) we used 5% P value as our rule of thumb. We would have taken the one study as something to act on and the other one as interesting but in need of further analysis if it was important enough. So For some groups the difference in confidence would be significant.