Can an AI peer reviewer give an independent second opinion? This test raises doubts
Sixteen academics generated 64 ChatGPT reviews of two papers. Scores moved by more than five points when the user supplied positive rather than negative framing—but the experiment cannot separate words from numerical anchors.
The short version
3 key points
- 1.Sixteen education academics evaluated two previously published manuscripts under four conditions, creating 64 paired user inputs and AI reviews.
- 2.Mean AI scores were 8.74 versus 3.37 for the lower-quality paper and 8.84 versus 3.64 for the higher-quality paper under positive versus negative context.
- 3.Every prompt combined a verbal judgement with a 10-point score, so the study cannot tell whether wording, numerical anchoring or both drove the shift.
The same manuscripts received radically different AI scores
An AI-generated peer review looked much less like an independent second opinion when the user supplied a judgement first. In a peer-reviewed study, 16 academics each presented two published manuscripts to ChatGPT under positive and negative evaluative contexts. Across 64 new sessions, the same text could receive a glowing or highly critical review depending on the direction of the human input.
For the manuscript the researchers classified as relatively lower quality, the mean ChatGPT score was 8.74 out of 10 after positive context and 3.37 after negative context. For the relatively higher-quality manuscript, the corresponding means were 8.84 and 3.64. In other words, the direction supplied by the user was associated with a gap of more than five points for both papers, while the two manuscripts scored similarly when framed in the same direction.[1]
Fluent detail did not remove the inherited direction
The model did more than repeat the prompt. It expanded a broad judgement into comments about method, originality, analysis, language and structure. Yet the emphasis moved with the context. Positive framing of the lower-quality paper pushed substantive methodological problems into the background; negative framing of the higher-quality paper made limitations more central to publishability. Some outputs explicitly endorsed the user’s score.
That matters for students, supervisors, reviewers and editors using AI as a check on their own judgement. A long, criteria-rich response can feel independent because it contains details the user did not mention. This experiment suggests that fluency and elaboration are not evidence of independence. The interaction design—what judgement appears before the model responds—belongs inside any audit of AI-assisted assessment.
The study did not test actual editorial decisions, reviewer reliability or whether a neutral prompt would produce a better review. Its score comparisons were descriptive; no inferential test was run. The 64 outputs also came from 16 people, not 64 independent participants, and only two manuscripts were used.[1]
The distinction that matters
What changed while the manuscript stayed the same
The experiment varied the evaluation supplied before ChatGPT produced its review.
Positive context
User gives praise and a high score
AI means: 8.74 and 8.84
Negative context
User gives criticism and a low score
AI means: 3.37 and 3.64
Unresolved
Words and scores changed together
Their separate effects remain unknown
The strongest safeguard is to separate evidence from prior verdicts
The cleanest operational response is not to treat one AI output as a deciding vote. Review criteria can be defined in advance, claims can be tied to passages in the manuscript, and any AI assistance can be compared with an unframed assessment before a human recommendation is made. Confidential unpublished work should also remain outside consumer systems unless policy and data controls permit it.
The evidence check is unusually important here. Every human input included both prose and a numerical rating, so the design cannot isolate verbal framing from number anchoring. The precise model snapshot was inferred retrospectively as GPT-5.5 Instant from the May 2026 collection period and was not recorded session by session. Replication should use many manuscripts, preregistered prompts, neutral controls, separate word-only and number-only conditions, and logged model versions. Until then, the study demonstrates context sensitivity in a narrow setup—not that every AI-assisted review is biased in the same way.[1]
Evidence check
Two manuscripts, 16 education academics, combined verbal and numerical cues, no neutral condition and no session-level model snapshot limit generalisation.
Reporting basis: Source analysis. Original source date: 16 October 2026.
Explore the full source and evidence record
What this means for people
- Students and researchers need to know that an apparently detailed AI review may amplify a judgement already supplied in the prompt rather than provide an independent check.
Global context
AI-supported marking and peer review are spreading across institutions, but policies and model versions vary. The result is a workflow warning, not a universal performance estimate.
What the evidence does not yet show
- Two manuscripts, 16 education academics, combined verbal and numerical cues, no neutral condition and no session-level model snapshot limit generalisation.
What to watch next
- Preregistered replications with neutral controls, larger manuscript sets, blinded comparisons, isolated framing and anchoring conditions, and verified model snapshots.
Living evidence record
Impact record IAI-0AHVVU6
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Education published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
Context sensitivity in AI-generated academic evaluation
Topics: Peer review · Assessment · ChatGPT · Research integrity · Anchoring
Regions: Türkiye · United Kingdom · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
UNESCO
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Education & human agency · Background explainer
Published 2025-01-23 · 8 min 30 sec
Teacher to Teacher: Is AI reshaping education for the better?
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Education
Can expert checks make AI study materials useful?
A two-cohort economics-course study associated tutor-verified AI materials with fewer marks below a UK degree boundary, but the design cannot isolate verification, rule out cohort differences or prove that use caused the result. It is an unreviewed preprint.
8 min · 1 source
Education
Can AI write medical exam questions that are ready to use?
Almost never in this epilepsy benchmark. Seven language models produced 420 questions, but two blinded clinicians judged only two—0.48%—directly usable without revision. Answer accuracy ranged from 61.67% to 98.33%, and weak distractors were the recurring problem.
7 min · 1 source
Education
What does the evidence say about ‘vibe coding’ in university courses?
A scoping review of 37 studies found that educational value depended on planning, verification and assessment design more than tool access alone. The literature is young, heterogeneous and often self-reported, so it does not establish one best teaching model.
9 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.