First-time reviewer
Posted by Jeremy on November 24, 2009 · 3 Comments
This year I’m getting to do something fairly new to me: review other peoples’ research.
Actually, it’s totally new. I’m ashamed to say I never reviewed papers for the ICA conference as a graduate student. I know it’s the closest thing grad students have to a civic duty, but asking a grad student to conjure up free time at the end of November …. well, you might as well ask me to make fat/sugar free ice cream taste good.
Yes, other grad colleagues found time to do it. I am aware of this. They are phenomenal humans.
This year I volunteered for ICA and gave myself a bit of penance for my past sins by volunteering to do reviews for three divisions. So it was pure joy to have 13 papers dropped on me in the second week of November with the oh-by-the-way message that they were due by December 7. By the time Thanksgiving Day rolls around, I hope I’m not serving a paper on social capital theory and reviewing cornbread stuffing.
Anyhow, despite the sarcasm I do think reviewing is an interesting process. It’s always interesting to see how others view a research problem, perhaps even a problem that you’ve thought about yourself. I find myself mostly struggling with how to assess the writer’s use of methods. At times I find myself reading and thinking the writer chose the wrong way to conduct their experiment/survey, or that their data analysis procedure was wrong. I want to correct them, but I keep coming back to the words of Glenn Leshner from my Advanced Quantitative Methods course at Missouri: Did they do the study they said they would do? You can’t penalize them for not doing it the way you’d do it. You can, however, offer suggestions.
So I can do that. ICA uses a 1-7 rating system across several categories, and that’s straightforward enough. The kicker becomes that bottom question, as to whether you recommend acceptance or rejection. That’s when the questions above start to churn. Sure they did the study they said they’d do, but if the method doesn’t answer the question well or ignores valuable parts of the question, is it good research? If they are doing exploratory research in an area of study that is beyond that in terms of the literature, is it good? That one is a little harder. Most of the research that is tougher to review is lacking because the review of past research is so lacking; they end up trying to reinvent the wheel, and it seems to detract from the findings.
Those are the tough ones, but some of the papers are fun to read. When I get my hands on a well done paper I feel like I’m getting a sneak peek at something that is pushing the boundaries of knowledge. I find it’s really hard to craft intelligent feedback on papers you really like as a reviewer; as a grad student it’s easy because you just go fanboy. As a reviewer, you have to say something smart.
This is your first time doing ICA and you wound up with 13? Double ugh. I’m just about through my nine, and it’s a more impressive ratio so far than in past years. Have you found it getting easier as you’ve gone? Every year it seems I have to remember how to read these things efficiently, but it’s usually a fun time.
I’ve got four to go, and I’m not sure if that means I was more or less productive over the break than I should have been. I saved the ones due 12/8 for last. It’s gotten easier only because I’ve had to get some under my belt and see the wide range of paper quality. The hardest part for me is judging whether to recommend for acceptance or rejection. In a set of four, there usually is one good one and one bad one and a couple in various parts of the middle. But even if I rank one of the middle papers above the other, does that mean give it the thumbs up on acceptance?
My method at this point is to score it, and if it has a good score and I think it’s the kind of paper that furthers scholarship, thumbs up. But I’m feeling less bad about any papers I’ve had rejected despite good reviews. It’s hard to figure sometimes.
Not necessarily on the acceptance question. I’ve had both fairly well-ranked papers that I’ve rejected and more mediocre rankings that I’ve accepted. For me, that final thumbs up/down is decided on the basis of originality and potential (and appropriateness to the division, but that’s usually a scored variable). Also, I’ve whacked at least one because the spelling, editing and style were just so terribly heinous – if you do clever work but don’t care enough to make it readable, don’t expect the poor abused volunteer reviewer to go to bat for you.
For me, the accept/reject question is my chance to make a qualitative decision (sounds funny for a binary choice): Regardless of the numbers, do I think this deserves a chance? The nice thing about the system is that, if I do occasionally drop the ball on the gut-reaction side, there are both my quantitative scores and other anonymous reviewers to balance my moment of foolishness.