A news story reports that a new “study” has established a link between a driver’s chance of getting into an auto accident (or getting a traffic ticket) and the driver’s astrological sign. Lee Romanov, president of insurancehotline.com, looked at 100,000 drivers, and their tickets and accidents, as well as their astrological signs. Ms. Romanov reports that, “The results are overwhelming, showing drivers of certain astrological signs are prone to getting more tickets, while others seemed destined to have accidents.” Ms. Romanov has written her findings in a book (you can buy it throught the website) called Car Carma.
It’s all too easy for non-statisticians to reach invalid conclusions about the numbers; I know how easy it is for actual statisticians to go astray! So I’d love to check the statistical validity of her conclusions. Unfortunately, neither the news story nor the website gives any numbers other than “100,000 drivers.” That’s not much to go on. But I’m not really that concerned with the so-called “study.” I’m more interested in the statistics.
Since I have only one hard number from Romanov — I’ll make some up. Let’s be perfectly clear, this is not about Romanov’s conclusions (I’d need the actual numbers to discuss that), it’s about what can happen when good numbers go bad.
Let’s begin by assuming as a null hypothesis that there’s no correlation between astrological sign and likelihood of a traffic accident. With a total of 100,000 in the “study” of drivers, and 12 astrological signs, we have about 8,333 drivers per astrological sign. Let’s assume that there are, in fact, exactly 8333 for each sign of the zodiac (so there are actually only 99,996 in the whole sample).
According to data from the Insurance Information Institute website, the fraction of drivers involved in traffic accidents in 2004 in the U.S. was 9.7%, with only 0.032% involved in fatal accidents (I consider these numbers to be reliable). Let’s adopt these as representative of Canada as well as the U.S. (Ms. Romanov’s company is Canadian). Then for each astrological sign, we would expect — if there’s no real correlation with astrological sign — about 808 drivers will be involved in an accident, of which about 2.7 will be involved in fatal accidents. These are the numbers we expect, and the expected value is referred to as the mean. We can also compute the uncertainty in these expectations. This is usually expressed as the standard deviation, which is the root-mean-square of the likely difference from the mean. For total accidents, the standard deviation is 27, for fatal accidents it’s 2.7. The “normal range” for what we will actually observe is generally about plus or minus twice the standard deviation. This is the range (based on the normal distribution) which will encompass the result 95% of the time; only 5% of results will normally fall outside this range. This is the usual standard for statistical tests in science, referred to as “95% confidence,” or “5% false-alarm probability.”
For accidents in general, and for a single astrological sign, we expect about 808 +/- 54, or somewhere between 754 and 862. For fatal accidents, we’d expect somewhere between -2.7 (yes, that’s negative) and 8.1. But these latter figures are misleading, because the numbers are so small we can’t apply the “normal” distribution even as a good approximation (we should apply the binomial distribution, but I’m not interested in being overly rigorous, I just want to get some good ballpark estimates, so I’ll approximate it by the Poisson distribution). We actually expect between 0 (zero) and 6 fatal accidents per astrological sign.
But in fact we can expect even more variation that that. That’s because there are 12 astrological signs, so we’ll actually be looking at the results of 12 independent tests. That gives us not just one chance to get an extreme result, we have 12 chances! So the expected range for the most extreme results will be about 731 to 885 for total accidents, and 0 to 8 for fatal accidents. And that’s the range for 95% confidence; there’s a 5% chance of one of the numbers being even more extreme.
I can well imagine Ms. Romanov noticing that drivers from one of the astrological signs had 885 accidents (out of 8,333 drivers) while another sign experienced only 731 (out of 8333). The two groups are the same size, but the first group had 21% more accidents! I can easily imagine that she would consider this result “overwhelming.” In fact, I can easily imagine that a lot of people would. But in truth, these numbers aren’t much outside the range we expect to get just from the random nature of statistics (and actually I’ve played fast-and-loose with the analysis myself — I’m just making a point). Likewise, with one group experiencing 8 fatal accidents while another group had none, the idea of an “overwhelming” result is all to easy a conclusion to jump to.
I’m oh-so-skeptical about Ms. Romanov’s findings from a statistical point of view. But one thing I think is highly likely: she’s one hell of a marketing genius. After all, the story of her “study” is a big headline on msn.com, and here I am writing about it on my blog. In fact the first dozen or so times I tried to view the website I got an error, probably because traffic on her website is “through the roof.” I’ll go out on a limb, and make a prediction based on no hard numbers at all: she’s gonna sell a lot of books.