In January 2019, I started my term of service as associate editor of the Journal for the Education of the Gifted. It’s a niche journal, but one that I’m proud to be part of. The editors, Dr. Jennifer Jolly and Dr. Jennifer Robins, are amazing at what they do, and it’s great to be part of their team.
This week, I got feedback from a peer reviewer for a manuscript that I’ve supervising at JEG. (That’s what all the cool kids call the journal.) The reviewer stated that she wanted changes to the manuscript, but there were no comments to guide the authors of what changes were needed. It was a very unhelpful review.
So, to prevent that from happening again, here are some tips for reviewers. My main experience is in social science reviewing, but people in other fields may find these tips helpful.
- Start with the assumption that the manuscript can be publishable. For the manuscript to get to a journal, someone thought it was worth the time to create. Every manuscript deserves a fair chance. Rejecting a manuscript should be reserved for two scenarios. The first is if a study is so bad that it cannot ever become publishable. This occurs when the logic has inherent, irredeemable flaws, or if the study design is so bad that it cannot answer any nontrivial scientific questions. The second scenario warranting rejection is if a paper would need such an intensive rewrite that it is not worth the journal editor’s and reviewers’ time to shepherd the manuscript to publishable quality. In that case, a rejection is best so that the author can take the manuscript elsewhere after a rewrite.
- Tell the author what is wrong and how to fix it. Be specific. If a sentence needs rewording, suggest how it should read. It a paragraph or a page needs a rewrite, state what the message of the revision should be. If there is a methodological problem, explain which re-analysis of data (if any) can remedy it. Don’t make the author guess. They’re not a mind reader.
- Critique the paper that you’re reading, not the paper you wish you were reading. Unless a paper is a registered report, then the study is done. Don’t redesign it. You may wish that the authors had different research questions, but you are way too late to get the study changed. Just like a child, it is important to love a study for what it is, not for what you hope it could have been.
- Don’t judge by irrelevant criteria. Manuscripts should be accepted if they make a nontrivial contribution to the scientific conversation. If the methodology, logic, and writing quality are sufficient to meet this level of contribution, then the manuscript should be accepted. Irrelevant considerations, such as the novelty of the study, the prestige of the authors and/or their university, whether a study is subjectively interesting, or the political implications of a study should not factor into a review.
- Declare your biases. Every scientist thinks that some research is more important, better, or more valuable than other research. I have a bias in favor of international research because I think that psychology needs to learn more about non-Western populations. On the other hand, I do not care for research on people’s opinions about an experience (e.g., teacher perceptions of professional development experiences). Declaring these sorts of biases helps reviewers be more fair to a manuscript, and it helps an editor make an informed decision about the manuscript’s fate.
- Have an eye for detail . . . Mistakes and shoddy research get published when no one involved—editor, reviewers, authors—is attentive. Check that statistics are mathematically possible, that the sample size is consistent, and that effect sizes were calculated correctly. This is basic stuff that is easy to verify. Increasingly, I think that reviewers should ask for data and computer code/syntax to verify the analyses are reported correctly.
- . . . but don’t proofread. Publishers have full-time copy editors that are paid to fix grammar, spelling, punctuation, and other basic errors.
My colleague, Dr. Kate E. Snyder (2018), has some practical suggestions about the process of reviewing a manuscript and some additional helpful advice.
Good luck reviewing! It’s a valuable service, but only if reviewers do their job well.
Snyder, K. E. (2018). How to be a more effective reviewer. Gifted Child Quarterly, 62, 251-254. doi:10.1177/0016986218754495