Here's What We're Doing -- See blog "Statistical Heartburn and Long-term Lessons"
Good discussion on this post. Here are two key clarifications to make about data analysis and about the stressed-out workloads of post-docs.
P-hacking and MTurk-iterating isn’t helpful to science, and it’s one of the reasons our lab seldom cites on-line studies. However, P-hacking shouldn’t be confused with deep data dives – with figuring out why our results don’t look as perfect as we want.
With field studies, hypotheses usually don’t “come out” on the first data run. But instead of dropping the study, a person contributes more to science by figuring out when the hypo worked and when it didn’t. This is Plan B. Perhaps your hypo worked during lunches but not dinners, or with small groups but not large groups. You don’t change your hypothesis, but you figure out where it worked and where it didn’t. Cool data contains cool discoveries. If a pilot study didn’t precede the field study, a lab study can follow -- either we do it or someone else does.
About Post-doc workloads. Academia is impatient for publications. It’s the reason why most professors don’t get tenure at their first school (I didn’t get it until my 3rd school). For Post-docs, publishing is make-or-break – it determines whether they stay in academia or they struggle in academia. Metaphorically, if they can’t publish enough to push past the academic gravitational pull as a post-doc, they’ll have to unfairly fight gravity until they find the right fit. Some post-docs are willin to make huge sacrifices for productivity because they think it's probably their last chance. For many others, these sacrifices aren’t worth it.
What follows is a tale of two young researchers.
There’s been some good discussion about this post and some useful points of clarification and correction that will be made with these papers. All of the editors were contacted when we learned of some of the inconsistencies, and a non-coauthor Stats Pro is redoing the analyses. We’ll publish any changes as erratum (and we’ll have an analysis script). This will also give us a chance to change oversights such as cross-citing the papers (they all came out within a year of each other, which led to that slipping between the cracks. Sorry.)
Sharing data can be very useful – like with lab studies and large secondary data sets – and in some instances being willing to do so (or a good reason why not) is a precondition to publishing in some journals. When we collected the data for this study, our agreement to the small business and to the IRB would be that it would be labeled as proprietary and would not be shared because it contained data sensitive to the small town company (sales data and traffic data) and data sensitive to the small town customers (names, identifying characteristics, how many drinks they had, the names of the people they were sitting with, and so on). This is data that cannot be depersonalized since sales, gender, and companions were central to some analyses. (We had explained this when someone requested the data.) At the time we published these papers, none of the journals had the policy of mandatory data sharing, or we would have published these papers elsewhere.
Upon learning of these inconsistencies, we contacted the editors of all four journals to swiftly and squarely deal with these inconsistencies. We told them that we would have the data reanalyzed, and we would write addendums.
Importantly, this field study was not intended to test specific, pre-registered hypotheses. Instead it was intended to learn initial answers to some interesting unanswered questions about eating in a real life restaurant. Does your first bite of a meal influence your attitude more than your last bite? Do you regret eating expensive food less than eating cheap food? Does the person you’re eating with influence how much you eat or like the food?
A PhD econometrician (not a coauthor) is now reanalyzing the data to confirm or refute the published results, and they will be sent to the journal editors. Following this, we will make the addendums, the data analysis scripts, and the data (if we can sufficiently anonymize it to protect our research subjects) available at a link we will add here.
In the end, I think the biggest contribution of bringing this to attention (van der Zee, Anaya, and Brown 2017) will be in improving data collection, analysis and reporting procedures across many behavioral fields. With our Lab, a rapidly revolving set of researchers, lab and field studies, and ongoing analyses led us to be sloppier on the reporting of some studies (such as these) than we should have been. This past Thursday we met to start developing new standard operating procedures (SOPs) that tighten up field study data collection (e.g., registering on trials.gov), analysis (e.g., saving analysis scripts), reporting (e.g., specifying hypo testing vs. exploration), and data sharing (e.g., writing consent forms less absolutely). When we finish these new SOPs (and test them and revise them), I hope to publish them (along with implementation tips) as an editorial in a journal so that they can also help other research groups. Again, in the end, the lessons learned here should raise us all to a higher level of efficiency, transparency, and cooperation.
A PhD student from a Turkish university called to interview to be a visiting scholar for 6 months. Her dissertation was on a topic that was only indirectly related to our Lab's mission, but she really wanted to come and we had the room, so I said "Yes."
When she arrived, I gave her a data set of a self-funded, failed study which had null results (it was a one month study in an all-you-can-eat Italian restaurant buffet where we had charged some people ½ as much as others). I said, "This cost us a lot of time and our own money to collect. There's got to be something here we can salvage because it's a cool (rich & unique) data set." I had three ideas for potential Plan B, C, & D directions (since Plan A had failed). I told her what the analyses should be and what the tables should look like. I then asked her if she wanted to do them.
Every day she came back with puzzling new results, and every day we would scratch our heads, ask "Why," and come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually we started discovering solutions that held up regardless of how we pressure-tested them. I outlined the first paper, and she wrote it up, and every day for a month I told her how to rewrite it and she did. This happened with a second paper, and then a third paper (which was one that was based on her own discovery while digging through the data).
At about this same time, I had a second data set that I thought was really cool that I had offered up to one of my paid post-docs (again, the woman from Turkey was an unpaid visitor). In the same way this same post-doc had originally declined to analyze the buffet data because they weren't sure where it would be published, they also declined this second data set. They said it would have been a "side project" for them they didn't have the personal time to do it. Boundaries. I get it.
Six months after arriving, the Turkish woman had one paper accepted, two papers with revision requests, and two others that were submitted (and were eventually accepted -- see below). In comparison, the post-doc left after a year (and also left academia) with 1/4 as much published (per month) as the Turkish woman. I think the person was also resentful of the Turkish woman.
Balance and time management has its place, but sometimes it's best to "Make hay while the sun shines."
About the third time a mentor hears a person say "No" to a research opportunity, a productive mentor will almost instantly give it to a second researcher -- along with the next opportunity. This second researcher might be less experienced, less well trained, from a lessor school, or from a lessor background, but at least they don't waste time by saying "No" or "I'll think about it." They unhesitatingly say "Yes" -- even if they are not exactly sure how they'll do it.
Facebook, Twitter, Game of Thrones, Starbucks, spinning class . . . time management is tough when there's so many other shiny alternatives that are more inviting than writing the background section or doing the analyses for a paper.
Yet most of us will never remember what we read or posted on Twitter or Facebook yesterday. In the meantime, this Turkish woman's resume will always have the five papers below.