Open Mind

Good Numbers Gone Bad

January 22, 2007 · 9 Comments

A news story reports that a new “study” has established a link between a driver’s chance of getting into an auto accident (or getting a traffic ticket) and the driver’s astrological sign. Lee Romanov, president of insurancehotline.com, looked at 100,000 drivers, and their tickets and accidents, as well as their astrological signs. Ms. Romanov reports that, “The results are overwhelming, showing drivers of certain astrological signs are prone to getting more tickets, while others seemed destined to have accidents.” Ms. Romanov has written her findings in a book (you can buy it throught the website) called Car Carma.

I’m skeptical.


It’s all too easy for non-statisticians to reach invalid conclusions about the numbers; I know how easy it is for actual statisticians to go astray! So I’d love to check the statistical validity of her conclusions. Unfortunately, neither the news story nor the website gives any numbers other than “100,000 drivers.” That’s not much to go on. But I’m not really that concerned with the so-called “study.” I’m more interested in the statistics.

Since I have only one hard number from Romanov — I’ll make some up. Let’s be perfectly clear, this is not about Romanov’s conclusions (I’d need the actual numbers to discuss that), it’s about what can happen when good numbers go bad.

Let’s begin by assuming as a null hypothesis that there’s no correlation between astrological sign and likelihood of a traffic accident. With a total of 100,000 in the “study” of drivers, and 12 astrological signs, we have about 8,333 drivers per astrological sign. Let’s assume that there are, in fact, exactly 8333 for each sign of the zodiac (so there are actually only 99,996 in the whole sample).

According to data from the Insurance Information Institute website, the fraction of drivers involved in traffic accidents in 2004 in the U.S. was 9.7%, with only 0.032% involved in fatal accidents (I consider these numbers to be reliable). Let’s adopt these as representative of Canada as well as the U.S. (Ms. Romanov’s company is Canadian). Then for each astrological sign, we would expect — if there’s no real correlation with astrological sign — about 808 drivers will be involved in an accident, of which about 2.7 will be involved in fatal accidents. These are the numbers we expect, and the expected value is referred to as the mean. We can also compute the uncertainty in these expectations. This is usually expressed as the standard deviation, which is the root-mean-square of the likely difference from the mean. For total accidents, the standard deviation is 27, for fatal accidents it’s 2.7. The “normal range” for what we will actually observe is generally about plus or minus twice the standard deviation. This is the range (based on the normal distribution) which will encompass the result 95% of the time; only 5% of results will normally fall outside this range. This is the usual standard for statistical tests in science, referred to as “95% confidence,” or “5% false-alarm probability.”

For accidents in general, and for a single astrological sign, we expect about 808 +/- 54, or somewhere between 754 and 862. For fatal accidents, we’d expect somewhere between -2.7 (yes, that’s negative) and 8.1. But these latter figures are misleading, because the numbers are so small we can’t apply the “normal” distribution even as a good approximation (we should apply the binomial distribution, but I’m not interested in being overly rigorous, I just want to get some good ballpark estimates, so I’ll approximate it by the Poisson distribution). We actually expect between 0 (zero) and 6 fatal accidents per astrological sign.

But in fact we can expect even more variation that that. That’s because there are 12 astrological signs, so we’ll actually be looking at the results of 12 independent tests. That gives us not just one chance to get an extreme result, we have 12 chances! So the expected range for the most extreme results will be about 731 to 885 for total accidents, and 0 to 8 for fatal accidents. And that’s the range for 95% confidence; there’s a 5% chance of one of the numbers being even more extreme.

I can well imagine Ms. Romanov noticing that drivers from one of the astrological signs had 885 accidents (out of 8,333 drivers) while another sign experienced only 731 (out of 8333). The two groups are the same size, but the first group had 21% more accidents! I can easily imagine that she would consider this result “overwhelming.” In fact, I can easily imagine that a lot of people would. But in truth, these numbers aren’t much outside the range we expect to get just from the random nature of statistics (and actually I’ve played fast-and-loose with the analysis myself — I’m just making a point). Likewise, with one group experiencing 8 fatal accidents while another group had none, the idea of an “overwhelming” result is all to easy a conclusion to jump to.

I’m oh-so-skeptical about Ms. Romanov’s findings from a statistical point of view. But one thing I think is highly likely: she’s one hell of a marketing genius. After all, the story of her “study” is a big headline on msn.com, and here I am writing about it on my blog. In fact the first dozen or so times I tried to view the website I got an error, probably because traffic on her website is “through the roof.” I’ll go out on a limb, and make a prediction based on no hard numbers at all: she’s gonna sell a lot of books.

Categories: mathematics

9 responses so far ↓

  • Steve Bloom // January 23, 2007 at 12:27 am

    Did the article quote her staff actuary Greg Rasputin? :)

  • Jim // January 23, 2007 at 1:24 am

    I’d be curious to know if she looked to see whether or not her results were statistically significant. That answer would reveal in short order just how “overwhelming” her results really are.

  • Peaseblossom // January 23, 2007 at 1:43 am

    Sometimes I wish was dumb enough to write a book like that and make a kajillion dollars…

  • tamino // January 23, 2007 at 2:04 am

    I’m also curious whether or not she removed the influence of other demographic factors that could skew the result. For instance, were most of the Libras young adults while the Leos were middle-aged?

  • Brian // January 23, 2007 at 3:38 am

    Nice post…enjoyed it.

    Plus, in the end, there’s always the old adage: ‘correlation does not imply causation’

  • tamino // January 23, 2007 at 4:00 am

    I checked out the list of worst drivers, accident-wise. I notice that the list, from worst to best (according to Romanov), is in this order (going by the month covered by most of the sign):

    October
    November
    January
    February
    April
    December
    March
    May
    September
    June
    July
    August

    I notice that autumn/winter birthdays dominate the “worst” list, summer birthdays dominate the “best” list.

    So here’s a crazy theory: a big factor in accidents is risky or impaired driving on one’s birthday. The drivers most susceptible would be those celbrating birthdays during autumn/winter, as driving conditions are more hazardous (remember the “study” is in Canada). So naturally, autumn/winter signs will have higher accident rates.

    Frankly, I’m skeptical about this too. But it’s an interesting thought.

  • Dano // January 23, 2007 at 4:23 am

    Note the number of months with short days up near the top. Fatigue, low sun angle making confusion, Seasonal Affective Disorder.

    Best,

    D

  • Yelling in the fog // January 23, 2007 at 12:09 pm

    Yahoo, my birthday is in June!! I can drive how I like and the gods of insurance will look out for me. Yeah baby!

  • WHEN YOUR BIRTHDAY ISN'T // March 18, 2007 at 10:59 am

    Here’s another possible ‘variable’ in Ms. Romanov’s study; I have a friend who, when her mother died and she was going through her Mom’s papers, found that her birthday was actually 6 wks. earlier. Her mom had given her a “birthday” of the day she had obtained her from an adoption agency. Can you imagine finding that out in your 30’s? So any accident report my friend filed would have placed her among Pices as opposed to Aquarius. In this case, Ms. Romanov’s base data is faulty.

Leave a Comment