Peer reviewing your first paper

Peer reviewing your first academic paper can be a daunting task, especially if you’re a newly minted Post-Doc, or an Early Career Researcher reviewing a manuscript written by more senior colleagues. Despite being an intimidating, and likely difficult, task acting as a peer-reviewer has several tangible benefits. More importantly, peer-review is part of your scholarly responsibility. By the time that you’re asked to review a paper for the first time, you will more than likely have published several of your own papers. Each of these papers will have required at least two referees to write reports, in addition to an editor - it’s time to give something back! In addition, the members of the journal's editorial board are likely to be highly respected academics in your field and, demonstrating expert subject knowledge, will elevate your profile and, potentially, lead to new collaborations, conference invitations, etc.

It’s likely that, when you are first invited to review a paper, you will have already been on the receiving end of a handful of referee reports; these will give you some idea on how to approach your report. Unfortunately, it is unlikely that all the reports you receive on your own papers will be exemplary, or even mediocre! Moreover, there are no courses on “How to peer review”. With this in mind, I’ve decided to jot down a few notes on how to approach your first peer review.

You will not feel like an expert

The chances are that, when you're invited to referee you first paper, you will have only carried out research on a very narrow part of your very specialized sub-field. Naturally, you will probably feel like you don't have sufficient expertise to make a judgement on another's work. Whilst this may be partially true, at least one other expert in the field (the editor) thinks that you know enough about the wider field to at least express an opinion.

Obviously, if you are clearly unqualified to review the paper, then you should politely decline the invitation. However, if the topic of the paper is relatively close to your own research, then I strongly suggest that you accept the invitation. Even now, very few of the papers that I am asked to review align exactly with the topics that I have worked on; and, if they did, they would likely be written by my collaborators (more about this later).

Don't pretend to be an expert

The other side of this coin is that you shouldn't pretend to be an expert on things that you don't know anything about. It is quite acceptable to only comment on the portions of the paper on which you have sufficient knowledge. You are not necessarily expected to be able to give an authoritative review on all aspects of the paper. The editor will have taken the balance of expertise into account when selecting the reviewers, with the other referee being an expert on the areas of the paper which are outside your research area.

As an author, there is nothing more infuriating that receiving referee reports from reviewers who clearly know nothing about the subject but, nevertheless, present themselves as an authority on the topic. Don't be that reviewer.

Don't assume that you're an idiot

The third side to this coin (😉) is that you shouldn't assume that you're an idiot. When you're at the early stages of your academic career it is tempting to assume that when you don't understand something in a paper or talk, it is your own fault. However, provided that you work in a closely related field, it is more than likely because the authors haven't explained things very well. Provided that you do so politely, you should highlight anything that you don't understand in the paper and ask the authors to consider restructuring their explanation.

It will take you a lot longer than you expect

After some practice, you should generally be able to produce a good quality referee report within 3-4 hours. However, for your first couple of papers, I would suggest giving yourself around two full-days to read the paper a few times and write your report.

Usually, I skim read the paper when first invited to review and decide whether I am qualified to review it, after which I respond to the invitation and put it away until I have some time to referee the paper. Then, I sit down and read the paper in its entirety without making notes or comments; this stage gives me an overall impression of the paper and I try to understand the broad strokes of the author's arguments and results.

Next, I carefully read through the paper making notes on the questions or corrections I find. I'll then look up any relevant literature that I need to, and skim read those papers. Finally, I sit down to write my report with the original paper and notes. Usually, I re-read the sections on which I have questions one more time to ensure that I have not miss-understood things.

The initial skim-read usually takes 5 minutes. The first full read-through takes me around 20 minutes, with the detailed read-through taking me around 1-1.5 hours. If necessary, I then spend some time reading the relevant literature. Writing the report usually takes me around half an hour to 45 minutes.

When accepting an invitation you should take note of the deadline by which the editor expect you to submit your report and make every effort to achieve this deadline. I am sure that you have experienced the long wait for overdue referee reports on your own papers and know how frustrating it can be. Don't be that referee!

Don't look for typos

If you spot any spelling or grammatical mistakes, then by all means, point them out. However, you are not a proof-reader; it is not your responsibility to go through the paper line-by-line looking for minor mistakes. Your primary role is to asses the scientific quality of the article and, unless the spelling, grammar, or style significantly affect the readability of the paper, then don't waste your time!

Your report

The structure of referee reports varies from field to field and the journal will likely give you some specific instructions, as well as some indication of the criteria on which you should review the paper. Generally, journals would like you to answer three main questions:

  1. Is the paper technically correct?
  2. Is the subject of the paper interesting and worthwhile investigating?
  3. Are the results of the paper new and likely to be of interest to the readership of the journal and the wider community?

You should already be familiar with the level of papers in your field. If you are reviewing for a journal that you are not familiar with, then it is always a good idea to take a look at a couple of papers from a recent issue so that you can judge their level and style.

Usually, I structure my reports as follows.

  1. A brief summary: Here you should outline the topic of research, the methods and the main results. Do not simply regurgitate the abstract - this is where you should demonstrate your subject knowledge and that you have understood the paper. The editor may also use this to judge how much weight to give your report in comparison with the other referee(s). Be concise and to the point; a few sentences should be enough, and certainly no more than a couple of paragraphs.
  2. List you main concerns: Here you should include any conceptual or technical problems that you have with the paper. Are the methods correct and appropriate, is the subject interesting, and the results new?
  3. Minor concerns: Here you should list minor corrections, such as missing referencing, typos, etc.
  4. Recommendation: Finally give a clear recommendation to the editor, usually you this recommendation will be one of:
  • Accept as it - the paper is essentially perfect and can be published in its current form
  • Minor Corrections - the authors should make some minor adjustments that do not affect the overall result or methods used.
  • Major Corrections - the paper needs significant work before it can be published. Significant problems with the methods, results, etc.
  • Reject - there is no hope of this paper ever being published. The methods are incorrect or inappropriate, the outcomes are not support by evidence, or the results are not new.

Be kind

First and foremost, you should be objective when reviewing a paper. However, it is also worth remembering that a group of people (and likely a single PhD Student/Post-Doc) spent many months working on this paper. And whilst you should be objective and give valid criticism, don't be nasty or personalize any mistakes in the paper. Stick to evaluating the work, don't project the work onto the authors, and avoid using terms like "the author(s)", "they", "them", "you", or "he/she".

Cast you mind back to any nasty or demoralizing referee reports that you received on your own paper. Try to avoid repeating the things that made you feel attacked or took as a personal criticisms. Don't be that reviewer!

Conflicts of interest and confidentiality

Before accepting an invitation to review, you should be confident in your ability to assess the paper impartially. You are extremely unlikely to be able to offer an impartial review of your PhD supervisor's paper if you only completed your PhD a few years ago, for example. Similarly, you cannot possibility be impartial about your current boss' paper. Other things to consider are whether you can give impartial reviews on a competing groups' work, or a colleague's paper etc. If you are in any doubt, then you should decline the invitation.

Finally it is likely that, at some point in your career, you will be asked to review a paper that is extremely close to one which you are working on, or a grant application that you are writing, or deals with a problem that you would like to be solved. Obviously, it is a huge breach of confidentially, etiquette, and professional standards, not to mention being incredibly nasty, to steal an idea from a paper that you are asked to review. I have no doubt that this sort of thing goes on in academia but, please, don't be that reviewer!

short signature.jpeg