Harvard Working Paper Finds Model Reviews Preferred Over Paid Pre-Submission Feedback

An accuracy audit against the manuscripts found no fabricated references in the web-enabled AI reports. The AI judge rated 5.5 percent of assertions concerning the manuscript incorrect, compared with 15.9 percent for human reports, roughly one-third as high. To test whether the judge could spot errors, the researchers inserted one true or one false sentence into each review. The judge identified every one of 258 inserted falsehoods and did not flag any of 258 inserted true statements. A replication with 416 comparisons, using Claude Opus and GPT systems in both reviewer and judge roles plus a different prompt and scoring rubric, produced AI win rates of 97.1 percent under an Opus judge and 94.2 percent under a GPT judge. [1]

Scholarly publishing uses peer review as its main quality filter. It serves two jobs: improving drafts and informing publication decisions. The paper says the traditional process struggles with delays, recruiting qualified volunteers and low agreement among experts reading the same manuscript. Human referees often miss major errors, give shallow comments or favor familiar methods and prestigious institutions. The working paper tests only the first job, improvement, not gatekeeping. [1]

Prior evidence is mixed. One 2026 BMJ Open analysis of 119 manuscripts found human reviewers made more detailed and varied points than ChatGPT. Another 2026 blinded cardiology-journal comparison of 40 manuscripts had two editors rate reviews from Claude, ChatGPT and Gemini above human reviews on five of seven criteria. Most earlier analyses concluded AI should supplement, not replace, human reviewers, the paper says. Other research warns about AI in scientific writing. An analysis of AI-generated research summaries found chatbots overgeneralized findings more often than human specialists; newer models generated as many as 73 percent more sweeping claims. An inquiry into AI-written mental-health literature reviews found nearly two-thirds of citations were invented or erroneous, especially on specialized topics. The paper notes the contrast with its zero-fabricated-source finding but distinguishes generating a review from critiquing an existing manuscript with web retrieval enabled. [1]

The authors outline policy options, not proven reforms. Journals could publish an official AI prompt matched to editorial standards and ask authors to submit the resulting review plus a letter responding to points of disagreement. A subsequent confidential model review could include checks for hidden prompt manipulation. Human referees would be reserved for difficult cases, while a human editor would retain the final publication decision. The authors also warn that broad reliance on similar models could push researchers toward mainstream arguments and make the literature more uniform. [1]

Kilian M. Pohl, a Stanford University professor not involved in the work, said in comments reported by PsyPost that the study shows model critiques can exceed human pre-submission feedback on several dimensions. He said it challenges the assumption that expert referees give more useful comments. But he cautioned that the authors measured constructive feedback, not the ability to spot pathbreaking research, detect subtle scientific mistakes or judge whether a paper fits a specific journal. Better-written and more complete reports should not be confused with superior scientific judgment, he said. [1]

George Currie, UK Research and Innovation’s senior strategy adviser for open research, who said he spoke for himself and was not involved, told PsyPost that peer review is costly, mostly unpaid, uneven in quality and does not reliably keep weak research out of the literature. Its role includes gatekeeping as well as improvement. He said AI’s appeal is not that it replaces a flawless process but that it might ease burdens and make the system more transparent while speeding communication and reuse of research. [1]

Gustavo A. Patino, an associate dean at Western Michigan University’s medical school, told PsyPost he expects AI review tools to enter journal workflows within about five years as models improve; that is a forecast, not an observed outcome. He said the paper emphasizes editorial benefits while overlooking what reviewing does for the wider research community, including exposure to ideas and scholars. The paper itself notes that it did not test whether manuscript revisions prompted by model feedback produce a stronger final article than revisions after human review. [1]

Eyal Aharoni, an associate professor at Georgia State University, told PsyPost that using a separate AI judge does not guarantee independence because models may query one another. He called the methods suitable for an initial investigation and said it would be worth developing the authors’ brief argument about justifications for keeping human review even when model review appears superior. He also called for follow-up studies, citing worries about dependence on a model evaluator. [1]

Michael R. Ladomery, a professor of genetics at the University of the West of England, told PsyPost that the absence of evidence on whether authors’ changes improved their papers was a major gap. He also noted that pre-submission reviewers may lack a specific journal’s context, whereas referees for a submitted paper are likely to judge it against that journal’s audience and standards. He argued that peer review is part of academic professional life and helps scholars sharpen critical skills, and he said its intellectual satisfaction and teamwork cannot be replaced by AI. [1]

The paper itself lists limitations: it did not measure revision outcomes; the blinded editor evaluation covered 10 manuscripts and 30 judgments rather than every category; and the human reviews came from a coaching service, not selective journal gatekeeping. Pohl said he has middling confidence in the findings, observing that the former editors were not necessarily specialists in every paper’s subject, so their ratings may have favored clarity and presentation over scientific substance. Much of the judging also relied on AI, which could favor AI prose. [1]

The headline results are descriptive: the paper reports preference rates and mean scores but no formal significance testing or interval estimates, and it has not been peer reviewed. That makes the findings a strong signal for further study, not a settled basis for replacing referees. The authors say future work should test whether AI feedback improves revised manuscripts and how journals can prevent authors from gaming published prompts. [1]

Pohl proposed a large prospective randomized study inside established journals comparing traditional human review, human review aided by AI, and review produced entirely by AI. Such a trial, he said, would test whether AI-assisted review yields better published science and more reliable editorial decisions without harming originality or accountability. Patino’s five-year adoption forecast and the authors’ workflow proposals remain scenarios or expectations, not evidence of what journals will do or what effects adoption would have. [1]

Questions to watch include whether peer-reviewed trials confirm the reported quality gap; whether journals measure revision outcomes rather than only review reports; whether safeguards against hidden prompt manipulation work; whether similar models make the literature more uniform; and whether concerns about reviewer training and editorial accountability lead to hybrid systems rather than replacement. For now, the evidence supports a narrow conclusion: in this pre-submission setting, AI critiques were preferred by an AI judge and a small panel of former editors, while outside experts disputed how far that result extends to journal gatekeeping and scientific judgment. [1]

Sources

PsyPost: Relationships

Prepared with AI assistance from the cited sources.