The Third Reviewer: An Editorial Decision Case

Please plan on 30–45 minutes to complete this exercise in one sitting.

You are the editor handling a manuscript undergoing peer review.

At each stage, make the decision you would make based only on the information available to you at that moment.

Explain your reasoning briefly.

Once you commit to a decision and move forward, you cannot go back and change your answer. New information will then be revealed.

There may not be a single correct answer.

Stage 1: The Late Reviewer

You are the editor handling a manuscript that was sent to three peer reviewers.

Reviewers 1 and 2 submitted their reviews on time. Both reviews are detailed, thoughtful, and useful. After reading them, you believe you have enough information to make an editorial decision.

Reviewer 3 has not submitted a review and is now two weeks past the agreed-upon deadline. The journal’s usual reminder notices have been sent, but Reviewer 3 has not responded.

You are ready to move forward with the manuscript.

Stage 1A: Three Reviewers

You receive a manuscript that you believe should be sent for peer review. You invite several potential reviewers.

Three reviewers accept the invitation. All three are given the same due date.

The first two reviewers submit their reviews well before the deadline. Both reviews are detailed, thoughtful, and useful.

You read them and believe you have enough information to make an editorial decision.

Reviewer 3's review is not yet due.

Stage 2: The Angry Reviewer

You decide that you have sufficient information from the first two reviews and cancel Reviewer 3's assignment before the due date.

You send Reviewer 3 a message explaining that two thorough reviews have already been received and that the review is no longer needed.

Reviewer 3 responds almost immediately—and is angry.

He tells you that he has already spent approximately three hours reviewing the manuscript and was just about to submit his review. When he tried to do so, he discovered that he had been locked out of the submission system.

He writes:

I agreed to review this manuscript and spent three hours doing it. Reviewers volunteer their time, and now that time has been wasted. My review is attached. Please remove me from your reviewer list.

What do you do now?

What, if anything, do you say to Reviewer 3?

Do you honor his request to be removed from the reviewer list?

Most importantly:

Do you read the attached review? Why or why not?

Stage 3: The Review

You decide to read Reviewer 3's attached review.

It is excellent.

It is detailed, thoughtful, professional, and well supported.

Reviewer 3 recommends rejecting the manuscript.

The first two reviewers did not recommend rejection.

More importantly, Reviewer 3 has identified what appears to be a major problem with the statistical analysis. Neither of the other reviewers mentioned it.

Reviewer 3 describes the problem as a “fatal flaw.”

If Reviewer 3 is correct, the problem may substantially affect the study findings and conclusions.

What do you do now?

Does Reviewer 3's recommendation change your editorial decision?

What additional information, if any, do you need?

Stage 4: Outside Your Expertise

You return to the manuscript and carefully examine the statistical analysis alongside Reviewer 3's comments.

You understand the concern.

You can see why Reviewer 3 has raised it, and the criticism appears plausible.

But statistics are not your area of expertise.

You are not confident enough to determine whether Reviewer 3 is correct.

You need more information.

Stage 5: The Decision Letter

You obtain additional evaluation of the statistical concern.

An additional statistical review agrees that there is a significant problem with the analysis, although the reviewer believes that the data could potentially be reanalyzed appropriately.

You also use an AI tool to examine the statistical issue. The AI assessment agrees, more or less, with Reviewer 3.

You decide to invite the authors to revise the manuscript.

Because you are following what you understand to be COPE guidance, you send the reviews to the authors exactly as they were submitted:

• Reviewer 1's review
• Reviewer 2's review
• Reviewer 3's review, recommending rejection and describing the statistical problem as a “fatal flaw”
• The additional statistical review
• The AI-generated assessment

Your editorial management system is confusing and makes it difficult to personalize decision letters.

You therefore use the journal's brief standard Revise template.

The letter asks the authors to revise the manuscript and respond to all reviewer comments.

You send the decision.

Stage 5A: The Phone Call

The corresponding author calls you.

She says she is confused about what the journal actually wants her to do.

Two reviewers are generally positive. Reviewer 3 recommends rejection and describes the statistical problem as a “fatal flaw.” The additional statistical review says the problem may be correctable. The AI assessment discusses the issue somewhat differently.

The author also tells you that she finds comments from one of the reviewers nasty, demeaning, and unprofessional.

She asks:

Which of these comments am I actually supposed to respond to? Do you want us to revise this paper, or have you already decided that the study is fundamentally flawed?

Also consider:

What, if anything, do you do about the review the author describes as nasty and unprofessional?

What do you do about the AI assessment?

Stage 6: The Revision

The authors submit a revised manuscript.

They have taken the statistical concern seriously. They reanalyzed the data, revised the results, and changed portions of the discussion and conclusions.

They provide a detailed response explaining what they changed.

The revised analysis appears to address the concern.

Now you have another decision.

Reviewer 3 previously asked to be removed from the journal's reviewer list.