Open Mind

Update

February 28, 2010 · 38 Comments

I’ve been working steadily on preparing the results of the GHCN analysis for publication.


First, I’ve been revising some of the programs so that they’ll be easier to run. Originally, a script which computed a grid average had to be run once for each grid, with the user manually inputting the grid coordinates. What a pain! Now I’ve got a routine which calls that function once for each grid, with the grid coordinates automatically supplied (from a reference file). So, all grids can be computed in a single run with no user interaction required. It does take a long time to compute! But I never said I was an efficient programmer.

I’ve also started writing the actual text of the paper, and creating some of the graphs which will constitute its figures. I’m really not sure how long the whole thing will take; that depends also on how much time I’m required to devote to other efforts. But I have put some things on hold until I get this done.

In the meantime, Anthony Watts continues to post articles which question the validity of the surface temperature record. Of note is this one, which contains this remarkable sentence:


Based on this station, alone, one can argue the USHCN data set is inappropriate for use as a starting point for other investigators, and fails to earn the self-applied moniker as a “high quality data set.”

One station.

Why on earth am I spending all this time and effort to analyze data from the entire GHCN, when I could have dismissed the surface temperature record using only one station?

Categories: Global Warming
Tagged:

38 responses so far ↓

  • thefordprefect // February 28, 2010 at 2:30 pm | Reply

    Please remember to remove test code – no commenting out. Don’t want more CRU hack code – do we!!!! :^)

  • Amoeba // February 28, 2010 at 2:32 pm | Reply

    Tamino, thanks for all the hard work.

    It’s hardly surprising that those obnoxious, snivelling little truds, tWatts et al. are whining, complaining and squirming, because after Menne et al. and your work, their ability to convince their zombie readership that unreality is reality and truth is lies is becoming increasingly challenged.

    It’s just another way of saying, ‘Don’t confuse me with the facts, my mind is already made-up!’

  • FredT34 // February 28, 2010 at 3:21 pm | Reply

    In Watt’s the Weatherman’s head, one station can prove all stations are wrong, one blogger can prove all science is wrong.

    He’s just noise.

  • Crispy // February 28, 2010 at 3:22 pm | Reply

    Having read the Schnare piece I gather his point is that if the adjustments made to this one rural weather station data points are invalid, then automatically the adjustments made to the rest of the data set can be called into question.

    A bit like atheists attacking the doctrine of biblical infallibility by finding small contradictions in the gospel accounts. :)

    He argues that USHCN and then GISS adjustments are done serially. He argues it is unreasonable to adjust a well-placed rural station for urban heat sink bias. He argues the adjustments lower the early temperature record to demonstrate an apparent warming trend in the later record. All of which fits the serious conspiracy theory over there at Wattsland.

    I’m trying to get my head around all this. I look forward to someone’s critique of his points, if not yours, Tamino. I know you’re flat chat on the paper.

    It is odd, reading the Watts blog. They remind me of Comical Ali, insisting all is well and hey, he might go ten pin bowling that night, while missiles fall behind him in Baghdad. They’re so damn confident nothing is happening, and yet the ice is melting, the migrations are shifting and we have the hottest decade on record. And it’s just getting started. Surreal.

    • Alex // March 1, 2010 at 7:26 pm | Reply

      “A bit like atheists attacking the doctrine of biblical infallibility by finding small contradictions in the gospel accounts.”

      As an atheist, I have to disagree with this comparison. If the Bible is claimed to be INFALLIBLE, then by finding one mistake in the Bible, it can’t be infallible.

      • Didactylos // March 1, 2010 at 8:14 pm

        *rolls eyes*

        If you are going to go off-topic and try to start a jihad, then at least don’t confuse infallibility, inerrancy and divine inspiration.

        As with climate and statistics, if this isn’t your field then do some research before inserting foot into mouth.

        Grumpy? Me?

      • Crispy // March 1, 2010 at 10:39 pm

        Granted, I should have typed ‘inerrancy’. It was 2 a.m. here.

        My point about Schnare’s debating tactic stands. He’s trying to bring the data set into question by attacking one data point. Wasting his time and ours, so it seems.

        Or perhaps he and Watts are working on their new book: “Debunking the Warmists – One Thermometer at a Time”.

  • Brian Angliss // February 28, 2010 at 3:39 pm | Reply

    Because you’re not intellectually bankrupt?

  • Crispy // February 28, 2010 at 3:45 pm | Reply

    Okay, re-reading your latest analysis it’s clear they’re wasting their time over at Watts place nitpicking individual station sites, since crunching the numbers on the raw data gives the same result as the adjusted data.

    Nevertheless, I’m curious about the justification for the serial adjustments. It seems to leave a flank wide open for criticism.

    Forgive my newbiness. Retaining this stuff is a challenge to the statistically unschooled.

    • Ray Ladbury // February 28, 2010 at 5:11 pm | Reply

      Crispy,
      The question is not what leaves one open to criticism, but what is right. If they didn’t make corrections, the denialists would scream just as loudly. Their goal is not understanding but self delusion.

      • Crispy // February 28, 2010 at 5:57 pm

        Well yes, but why do USHCN apply a UHI correction to a rural site? If it’s the right thing to do, is there an explanation of that bit of methodology somewhere? To the questing layman it does seem odd, and Wattsuponians play on that. Is there a discussion of USHCN and GISS weightings and adjustments somewhere? Am I a masochist, or what?

        [Response: The site (Dale Enterprise, VA) is listed as having population 31,000, not listed as a "rural area."]

      • carrot eater // February 28, 2010 at 10:02 pm

        There isn’t a separate UHI step in the current USHCN. The base homogenisation is meant to sniff out UHI trends.

  • Lamont // February 28, 2010 at 4:36 pm | Reply

    I think Watts has discovered that he can jump the shark and his audience wont notice…

  • Ray Ladbury // February 28, 2010 at 5:08 pm | Reply

    Anthony Watts: “One station!”

    In response to which, I have one word: MORON!

  • Robert // February 28, 2010 at 5:10 pm | Reply

    There was also Spenser’s recent re-analysis, which included this comment:

    “This [20% reduced warming in my data analysis] is a little curious since I have made no adjustments for increasing urban heat island (UHI) effects over time, which likely are causing a spurious warming effect, and yet the Jones dataset which IS (I believe) adjusted for UHI effects actually has somewhat greater warming than the ISH data.”

    This seems to dovetail nicely with the recent observation that the “UHI” has a net cooling effect (Menne 2010; http://www1.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/menne-etal2010.pdf). That the unadjusted data has less of a trend in his analysis might suggest that to him — he might at least be tempted to mention the paper. But, no. It’s further evidence they the measurements are all wrong. Classic conspiracy theory: disconfirming evidence = it’s even bigger than we thought!

  • Polderboy // February 28, 2010 at 6:35 pm | Reply

    Maybe this has been mentioned before but I can’t find it here or in the other posts dealing with D’Aleo and Watts.

    The weatherstations from the 70s and 80s weren’t “dropped off” in the 90s. The data of the weatherstations of the 70s and 80s was retroactively ADDED to the dataset in the last two decennia.

    Quote:
    “It’s common to think of temperature stations as modern Internet-linked operations that instantly report temperature readings to readily accessible databases, but that is not particularly accurate for stations outside of the United States and Western Europe. For many of the world’s stations, observations are still taken and recorded by hand, and assembling and digitizing records from thousands of stations worldwide is burdensome.

    During that spike in station counts in the 1970s, those stations were not actively reporting to some central repository. Rather, those records were collected years and decades later through painstaking work by researchers. It is quite likely that, a decade or two from now, the number of stations available for the 1990s and 2000s will exceed the 6,000-station peak reached in the 1970s.”

    From http://www.yaleclimatemediaforum.org/2010/01/kusi-noaa-nasa/

    • dhogaza // February 28, 2010 at 7:11 pm | Reply

      Exactly right, Polderboy. And I’ve read that there’s going to be an effort in the not-too-far-off future to do another retroactive fetch of data to add, though I don’t remember where I read it.

      However, “they dropped stations” sounds much more conspiratorial, and I’m sure Watts and D’Aleo know the truth and are knowingly lying.

      • carrot eater // February 28, 2010 at 10:11 pm

        NCDC has said there’ll be some more data in the upcoming new release.

        Between a few more stations and the new homogenisation method, there’ll be some minor differences in the record. So we can expect endless ‘blink’ graphs from the sceptics. I don’t understand their fascination with ‘blink’ graphs; just put both series on there at the same time.

  • Zeke Hausfather // February 28, 2010 at 6:41 pm | Reply

    Spencer’s result seems to emerge from comparing an adjusted dataset (CRUTem3) to a raw dataset (ISH) for lower 48 U.S. stations. You get almost the same result if you compare raw GHCN data (v2.mean) to adjusted data (v2.mean_adj) for U.S. stations: http://i81.photobucket.com/albums/j237/hausfath/Picture62.png

    The reason for these adjustments to U.S. stations are outlined here: http://www.ncdc.noaa.gov/oa/climate/research/ushcn/ushcn.html

    So while using the ISH dataset is rather novel, I’m not sure what all the fuss is about since we’ve known about the effects of USHCN adjustments to U.S. stations for a long time now :-p

    • carrot eater // February 28, 2010 at 10:06 pm | Reply

      Note also how his finding that the overall NH ISH raw matched the NH CRU adjusted didn’t draw any overarching commentary and conclusions. But then they look at the US, where we already know that adjustments make a difference, and that’s somehow proof everything is all wrong.

  • Zeke Hausfather // February 28, 2010 at 6:51 pm | Reply

    Tamino,

    Since grids are (usually) uniform, can’t you just do a for loop like:

    count = -87.5
    for latcoords = -90 (5) 90 do // the (5) designates to loop up in increments of 5
    latgrid = count if (stationlat >= latcoords and stationlat < (latcoords + 5)
    replace count = count + 5
    end

    for every station in the metadata file to assign it a 5×5 latitude grid? You could easily add an exception for latitudes near -90 and 90.

    [Response: I never claimed to be an efficient programmer. Besides, this wasn't meant to be production code, just a one-time job.]

    • Gavin's Pussycat // February 28, 2010 at 7:15 pm | Reply

      > this wasn’t meant to be production code, just a one-time job.

      Famous last words :-) …especially if a single run takes long, it is worthwhile picking the low hanging efficiency fruit — as you’re going to do more runs than you think now. In my experience.

  • Sekerob // February 28, 2010 at 7:08 pm | Reply

    Tamino In-Line to Zeke Hausfather,

    this one-time is not going to carry approval by McIntyred… he’ll demand easy readable code in soon R++ [Watt?, R++?, yes R++ which she has not mastered yet, her being several iterations behind reality]

  • dhogaza // February 28, 2010 at 7:13 pm | Reply

    Crispy:

    It is odd, reading the Watts blog. They remind me of Comical Ali

    That’s brilliant, love the analogy.

    Maybe if we just start calling him “Whackadoodle Watts” more people will catch on?

  • bsharp // February 28, 2010 at 7:32 pm | Reply

    Tamino, there are probably a few here willing to help with the code if you were to ask.

    [Response: After publication, anyone's free to modify it as they wish. But from a purely programming perspective, it's probably better to start from scratch.]

  • Robert // February 28, 2010 at 8:29 pm | Reply

    There was also Spenser’s recent re-analysis, which included this comment:

    “This [20% reduced warming in my data analysis] is a little curious since I have made no adjustments for increasing urban heat island (UHI) effects over time, which likely are causing a spurious warming effect, and yet the Jones dataset which IS (I believe) adjusted for UHI effects actually has somewhat greater warming than the ISH data.”

    This seems to dovetail nicely with the recent observation that the “UHI” has a net cooling effect (Menne 2010; http://www1.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/menne-etal2010.pdf).

    That the unadjusted data has less of a trend in his analysis might suggest that to him — he might at least be tempted to mention the paper. But, no. It’s further evidence they the measurements are all wrong. Classic conspiracy theory: disconfirming evidence = it’s even bigger than we thought!

  • Barton Paul Levenson // March 1, 2010 at 1:25 am | Reply

    I need help with a technical question.

    I need to do Granger causality analysis on a couple of climate variables. To choose the right lag period, one uses the Akaike Information Criterion (AIC), or a similar measure like the second-order AIC (AICc) or the Bayesian/Schwarz Information Criterion (BIC/SIC).

    But for finding if X causes Y or not, WHICH AIC (or whatever) do I minimize–the AIC for Y autoregressions, or for X? The dependent variable or the independent variable? I’ve been assuming the dependent variable is the one to choose the lag for, but is that correct?

    Say I’m looking at inflation (dP) and money-supply growth (dM). AIC is minimized at 10 years for dP and 4 years for dM. If I’m checking if dM Granger-causes dP, do I do the regressions for four years or ten years?

    Help!

  • David B. Benson // March 1, 2010 at 2:12 am | Reply

    Barton Paul Levenson // March 1, 2010 at 1:25 am — Does
    http://www.scholarpedia.org/article/Granger_causality
    help?

  • John McManus // March 1, 2010 at 5:45 pm | Reply

    Two things jumped off the Watts article. There are a number of ajustments made, not just one for UHI. Why do they intimate that the UHI is the only one. To the naked eye, steps appear in the data about 1920, 1935, 1960 and 1995.

    1995 saw a change in instruments, is expected and needs an ajustment. In 1932, the original record keeper died and his son took over. In 1960, he died and his dauhter took over. The most likely adjustment needed for these two occasions is TOB. The 1920 data step is unclear but I would guess a new shelter may have been installed.

    That was easy, even for a history major: no wonder the full list of ajustments was truncated.

    A blurb showed up on a bunch a blogs saying that no warming was detected for the US since 1895. The story from the New York Times read
    like the study was current but seaching further found the study was from 1989 and the data from 1987. Not an outright lie but certainly a lie of ommission.

    This is easy: every denial story contains one or more lies. When you know this and have some knowledge for scientific blogs , its easy to ferret ( sorry Stoat) them out.

    John McManus

    • carrot eater // March 1, 2010 at 6:08 pm | Reply

      For clarity: About half of the adjustments in USHCN v2.0 are not associated with events recorded in the station history. Station histories are necessarily incomplete, so the method looks for evidence of undocumented changes.

      As for the 1989 NYT article, in fairness to WUWT, it’s made quite clear that it’s an old article. The data is highlighted in red; the introductory discussion blurb is in that context.

  • John McManus // March 1, 2010 at 9:21 pm | Reply

    Thanks carrot eater:
    I see your point. WUWT is not one of the sites I looked at. I searched the newspaper aritcle, as close as I could find to the original paper and a bunch of websites. This is the first time I saw it clarified as 23 year old information.

    Do you know how to find what happened on the farm about 1920?

    John McManus

  • Barton Paul Levenson // March 2, 2010 at 1:54 pm | Reply

    DBB,

    No, it just mentions that one uses information criteria to determine the correct time period. Doesn’t say for which variable. Thank you for finding that site, though.

Leave a Comment