The strength of an experimental result is commonly stated in the form of a p-value. For example, if the p-value is 0.01, that means that if there were no effect, the probability of getting a result at least as extreme as the one actually obtained would be 0.01. This is a useful number, but it’s easily misinterpreted. What you really want to do is compare the probability of getting the exact result you got under the null hypothesis, to the probability of getting that result under the alternative hypothesis. The ratio of these two is called a Bayes factor. Only this information will allow you to rationally update your degree of belief in the null hypothesis. But from the p-value, you can’t tell.

I don’t know if this is widely known, but there’s a trick that you can use to translate from p-values to Bayes factors. Assume your prior distribution is symmetrical and peaked at the null hypothesis. Then the following formula gives you a minimum Bayes factor:

- e p ln(p)

For example, if your p-value is 0.05, your minimum Bayes factor is 0.4. That means the odds for the null hypothesis (that is to say, P(null true)/P(null false)) are multiplied by at least 0.4. A null hypothesis that starts out as 75% probable ends up being at least 31% probable. So a p-value of 0.05 isn’t nearly as bad as it sounds.

This article has a handy little table listing some other possible values. It also gives a weaker formula for a minimum Bayes factor that does not make any assumptions about the prior. This pdf article explains more about this minimum and about Bayes factors in general.

If you see a p-value quoted for a result you doubt (cough parapsychology cough), and if you know the basic technical concepts, it can be quite useful to have the formula or table at hand. Coincidental results are more common than you might think.