Serious flaws revealed in "longevity genes" study

When an article was published in Science last week reporting that DNA samples from exceptionally long-lived individuals differed detectably from those of normal individuals, it got plenty of positive attention from the mainstream media. However, the buzz from experts was rapid and telling: my colleagues in the statistical genetics community weren’t excited about the results, but immediately, profoundly skeptical.

For people who’ve spent years doing genome-wide association studies (GWAS), several things stand out as unusual from this paper: the very large effect sizes of the identified SNPs (as noted by colleague Jeff Barrett in the Guardian), the extraordinary claim that the identified variants were able to correctly classify individuals as potential centenarians with 77% accuracy (a totally unprecedented level of accuracy for a complex trait), the fact that the associated variants haven’t previously been associated with protection against any other common diseases (as you might expect them to be, given longevity is effectively a matter of avoiding or surviving every common disease), and several subtle technical issues, such as a rather strange-looking Manhattan plot (shown at the end of this post).

If the paper’s claims were true they would be truly remarkable. However, the general feeling from the GWAS community right now seems to be that the identified associations are likely to be largely or even entirely artefactual, the result of failing to fully control for differences in the genotyping methods used in the cases and controls. The study used a mixture of two different genotyping platforms (albeit both made by Illumina) for their centenarians, while the control data was taken from an online database containing samples examined using multiple platforms. Disturbingly, similar potential genotyping bias also affects their replication cohort.

In a great article in Newsweek today Mary Carmichael has a series of damning quotes from big-name geneticists casting doubt on the study’s findings. deCODE Genetics CEO Kári Stefánsson is (unsurprisingly) the most vociferous: he notes that there are consistent and previously known genotyping problems on the SNP chip used in the study for the two most strongly associated SNPs, and then goes further to argue that technical problems probably underlie nearly all of the reported associations in the paper:

Stefánsson says he is “convinced that the reported association between exceptional longevity and most of the 33″ variants found in the Science study, including all the variants that other scientists hadn’t already found, “is due to genotyping problems.” He has one more piece of evidence. Given what he knows about the 610-Quad, he says he can reverse-engineer the math in the BU study and estimate what fraction of the centenarians were analyzed with that chip. His estimate is about 8 percent. The actual fraction, which wasn’t initially provided in the Science paper, is 10 percent, the BU researchers tell NEWSWEEK. That’s close, given that Stefánsson’s calculations look at just two of the variants found in the study and there may be similar problems with others.

Carmichael goes on to note a major methodological flaw in the paper: the failure to even attempt to validate any of the associated SNPs on an independent platform. This is absolutely standard practice in normal GWAS, and should have been demanded by referees – especially given the extraordinary claims being made in the paper.

What needs to happen next? For a start, the authors should release the raw intensity data for their genotyping experiments, which would allow independent investigators to spot obvious problems. Doing so immediately on a public database would go a long way towards showing they’re not trying to cover up any methodological flaws. Ideally, they should also validate their putative associated SNPs using an independent platform and release those raw data as well.

More broadly, this is an important lesson for the increasing number of investigators wandering into the GWAS arena: they need to be aware that the genotype data they’re working with aren’t just clean, digital data points, but best-guess estimates (typically very reliable, but sometimes badly flawed) based on an noisy fluorescent intensity signal. There’s a reason why researchers working on GWAS spend so much of their time on a regimented series of upstream “data cleaning” steps and careful downstream validation of new associations – it’s all too easy for noisy data to introduce bias that produces a false association signal. So, kids, don’t end up in Newsweek for all the wrong reasons: talk to someone who really knows what they’re doing when it comes to GWAS data.

Finally, major journals need to stop letting sexiness push aside scientific rigor. Carmichael says it nicely:

Still, one has to wonder how the paper wound up in Science, which, along with Nature, is the top basic-science journal in the world. Most laypeople would never catch a possible technical glitch like this–who reads the methods sections of papers this complicated, much less the supplemental material, where a lot of the clues to this mystery were?–but Science‘s reviewers should have. It’s clear that the journal–which hasn’t yet responded to the concerns raised here–was excited to publish the paper, because it held a press conference last week and sent a representative to say as much.

If the key results from this paper do turn out to be based on easily-detected experimental artefacts, Science deserves to be embarrassed.

Anyway, here’s the image that really made me go “whoah” – the Manhattan plot from the paper, tucked away in the Supplementary Data, which shouts “artefact” to anyone who’s seen even a few GWAS papers. For the uninitiated, each dot in the plot represents a different SNP, with the alternating bands of colour showing different chromosomes. The y axis indicates the strength of the association between that SNP and longevity. The plot is unusual for a GWAS in that all of the highest-ranked SNPs are hanging out there by themselves, rather than being flanked by a column of other associated variants – a pattern characteristic of genotyping error rather than true association.

In contrast, here are the Manhattan plots from a “good” GWAS – the Wellcome Trust Case Control Consortium’s analysis of 7 different common diseases (I’ve trimmed out two uninteresting ones), with the statistically significant SNPs highlighted in green. You can see that basically all of the most strongly significant SNPs are found in a “tower”, the result of nearby SNPs being correlated with one another and thus all marking the same association signal:

Spot the difference?