Open Mind

Jury Duty

July 21, 2008 · 5 Comments

I got a request over the weekend to review a paper for a peer-reviewed journal. Such an effort can be interesting (you get to see new research before almost everybody else!), it can also be depressing (if the work is garbage, more so if it’s not that bad but just not “up to snuff”). But this much is certain: it takes time, and if you want to do it right (which I do) it can be a lot of work.


I try to keep in mind that it’s NOT the reviewer’s job to write the paper! It’s also not my job to make a recommendation based on whether or not I “like” the result. In this case I have some disagreements, but that won’t stop me from recommending acceptance. The data are sound, the methodology is appropriate (although there are some unaccounted-for statistical factors which will probably require a revision), and the result is likely to be useful (as well as interesting).

Scientists tend to be very busy — and that’s just counting the stuff we’re actually working on. All kinds of ideas are constantly arriving, and we’d love to pursue them, but there’s just not enough time to get them all. A hundred lifetimes wouldn’t be enough! So, the burden of review is definitely an imposition.

However: none of my papers would have been published if colleagues hadn’t taken *their* time to review them. I feel like we owe it to the scientific community to perform this service. I liken it to jury duty — if you want to reap the benefits of a free society, you need to be an active participant. Should I ever be accused of a crime (God forbid!), I’d want rational, intelligent citizens to sit on my jury — so I have to return the favor and do the same in return. It’s the same with review: if I want quality reviews of my work, with sufficient effort to give both a fair and a valuable recommendation, then I owe it to my colleagues to return the favor.

Often, reviewers suggest ideas that are extremely valuable, and it’s always worthwhile to get an opinion from someone who knows the field but isn’t so wrapped up in the topic at hand that it gets in the way of providing a “fresh viewpoint.” I’ve often thanked an anonymous referee for comments which greatly improved the quality of the manuscript — and I meant it every time.

So I took the job and I’ll actually work very hard on it, to give a top-notch review. Just like I always do. But I won’t complain; the time and effort I spend are more than repaid by reciprocal efforts on my behalf.

Categories: Global Warming

5 responses so far ↓

  • Hank Roberts // July 21, 2008 at 6:11 pm

    Why I respect the work you do enormously; why reading the actual science matters, summed up in two posts I left at RC recently:

    http://www.realclimate.org/index.php/archives/2008/07/weekend-round-up/#comment-92956
    http://www.realclimate.org/index.php/archives/2008/07/weekend-round-up/#comment-93007

  • Dave Andrews // July 21, 2008 at 9:15 pm

    Great, its always nice to know your expertise is appreciated and I’m sure you will do the job to the best of your ability.

    Wouldn’t have anything to do with the Huang, Mann etc paper mentioned by LB on Open Thread recently by any chance?

    [Response: No, it's not that paper. I'm honor-bound not to reveal any information about the author, subject, or even the journal, so I won't answer any more questions about any of that.

    This post is only an appeal to fellow scientists to shoulder the burden of review cheefully, for the good of all, and a little insight into the process for non-scientists.]

  • Swade // July 21, 2008 at 10:30 pm

    Your response brings me to an interesting question…. what actually delineates a “scientist” from a “non-scientist”?

    I’ve had the pleasure of reviewing papers for submission in my field before, but I really wouldn’t qualify myself as a scientist per se.

    I do statistics for a living, but wouldn’t call myself a statistician either.

    Is there a hard-and-fast rule for it? Can’t be a PhD or anything like that, or else Faraday would *still* be getting shafted :-) Although I guess he could be termed an “experimentalist”.

    Thoughts?

    [Response: There's no hard-and-fast rule, and no "perfect" answer. I'd say a scientist is someone who *does science* -- so there are a lot of Ph.D.s who don't make the cut! And there are many who aren't degreed, don't work in an institution, but in my opinion are definitely scientists because they're doing it. In fact, in astronomy there's a vast number of amateurs who do it only for the love of it (they don't get paid) but make very important contributions to science.

    If someone *claims* to be a scientist but I have reason to doubt, I'll check the peer-reviewed literature for publications as a primary indicator. But it's not infallible.]

  • henry // July 22, 2008 at 2:30 am

    Was the data and the code used available to you as a reviewer?

    Does the peer-reviewed journal require the data to be archived?

    I’ll wait for answers before asking any more questions…

    [Response: I've already said that I'll not answer any questions about the specifics of the paper, the author, or the journal. The work hasn't even been reviewed yet, let alone appeared in print, and I'm not only bound by my agreement not to disclose any such information, I'm bound by honor.

    It's also disgusting that people (you're far from the first) are trying to use a simple post extolling the virtue of willingly contributing to the process of peer review, into a criticism of scientific publishing/data archiving procedues. Your motives are transparent.

    You really aren't the first, and I only allowed this comment through to announce to all that your efforts are both disgusting and futile. There's nothing remotely controversial about this post, it's not even about climate science. Have you no shame?]

  • tamino // July 22, 2008 at 3:32 am

    I thought it would be a nice break from the constant bickering about global warming, to do a post not about global warming, just an appeal to scientists to do unrewarding work because we all benefit from such efforts.

    But a number of individuals want to use it as an excuse to rehash the issues of data archiving, code accessibility, journal requirements, and ultimately they’re aiming at the “hockey stick.” I’m pretty liberal with open threads, where that’s a recurrent theme for those who apparently don’t have anything better to talk about.

    But CLEARLY this post is not the place. I feel like I could do a post about the Boston Red Sox’ chances of winning the world series, and you’d try to turn it into a kangaroo court about the hockey stick.

    It truly is disgusting. Comments on this thread are closed.