Did AI editing help readers understand Chinese texts? Two experiments found no comprehension gain
Across 992 participants, target-guided rewrites did not improve author-keyed comprehension. One small alignment result was exploratory and tied to the authors' own scoring targets.
The short version
- What happened
- Study 1 randomly allocated 572 Chinese readers across original and GPT-4, Claude 3 Opus or Gemini Advanced versions; Study 2 allocated 420 readers across original, GPT-4 and Claude versions.
- Who it affects
- Writers, editors and readers should not assume that a polished AI rewrite is easier to understand or more faithful simply because it sounds smoother.
- What remains unproven
- Fixed Chinese texts and predominantly female, highly educated platform samples
Keep in mind
Fixed Chinese texts and predominantly female, highly educated platform samples
Study 1 sample extension followed inspection of interim results
Targets and answer keys guided both the rewrites and outcome scoring
No human-editor comparison
Author-written comprehension items were not independently validated
Product labels lacked model snapshots and sampling settings
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Target-guided rewrites changed one narrow signal, not comprehension
A peer-reviewed study tested whether AI-refined Chinese writing reached readers differently from the original text. Study 1 randomly assigned 572 participants—143 per condition—to original writing or versions refined by GPT-4, Claude 3 Opus or Gemini Advanced. Study 2 assigned 420 participants—140 per condition—to original academic excerpts or GPT-4 and Claude rewrites. Readers answered the same questions, and the models had been given the authors' intended ratings or answer-key information while revising.
GPT-4 produced a small improvement in exact-match emotional alignment in Study 1 (p = .002; signed r = .18). Gemini's unadjusted decline did not meet the corrected threshold, and Claude did not differ from the originals. More importantly, none of the systems produced a detectable comprehension difference. In Study 2, neither GPT-4 nor Claude changed author-keyed comprehension or the six designated scholarly attributes.[1]
The experiment does not settle whether ordinary AI copyediting helps
The design was more directed than everyday proofreading: models received the same target ratings or answer keys later used to judge readers. That can make a rewrite unusually sensitive to the outcome and creates a risk of target leakage. There was no human-editor comparison, so any difference cannot be attributed uniquely to AI. The comprehension questions were author-written rather than independently validated, and the participants were general readers rather than subject experts or peer reviewers.
Study 1 recruitment was expanded after interim results were inspected, so its final p-values are exploratory rather than confirmatory. The products were recorded only as GPT-4, Claude 3 Opus and Gemini Advanced, without model snapshots or sampling settings, limiting reproducibility. For writers and editors, the practical conclusion is modest: a target-guided rewrite did not reliably improve comprehension of these fixed texts. Independent, preregistered tests should compare AI and human editing, preserve version settings and assess meaning beyond answer-key matches.[1]
The distinction that matters
What changed across the two reader experiments
The experiments separated a narrow alignment score from whether readers recovered selected information.
Study 1 · 572 readers
Original plus three AI-refined conditions
One exploratory GPT-4 alignment gain; no comprehension difference
Study 2 · 420 readers
Original plus two AI-refined conditions
No detected alignment or comprehension difference
Human editing
No comparison condition
AI-specific effect unresolved
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring writing: compare the finding and its evidence limits.
Should AI wait until people think first?
In an unreviewed study of 398 adults across five countries, a writing assistant withheld direct generation until users stated a position and supporting argument. The design shifted effort earlier and improved one conditional error-classification measure, but did not significantly improve overall judgment accuracy.
Read the next reportExplore the full source and evidence record
What this means for people
- Writers, editors and readers should not assume that a polished AI rewrite is easier to understand or more faithful simply because it sounds smoother.
Global context
The experiments used Chinese-language materials and China-based platform participants. Results may not transfer to other languages, genres, professional editing workflows or newer model versions.
What the evidence does not yet show
- Fixed Chinese texts and predominantly female, highly educated platform samples
- Study 1 sample extension followed inspection of interim results
- Targets and answer keys guided both the rewrites and outcome scoring
- No human-editor comparison
- Author-written comprehension items were not independently validated
- Product labels lacked model snapshots and sampling settings
What to watch next
- Preregistered replications with fixed model versions
- Direct comparison with professional human editing
- Independent measures of factual fidelity, argument quality and voice
- Tests across languages, genres and specialist readers
Living evidence record
Impact record IAI-1XBUDA7
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
26 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Discover Artificial Intelligence published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 26 October 2026
Research question, topics and regions
Whether target-informed LLM rewriting changes author–reader alignment or comprehension of fixed Chinese texts
Topics: Writing · Communication · Editing · Research integrity
Regions: Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.