Can people reliably spot AI-written text?

What 390 adults were asked to judge

The analysis used data from a Japanese online survey conducted through Questant and the Macromill panel on 24 and 25 January 2025. The final sample included 390 adults aged 22 to 69, with a mean age of 47.8; 227 were men and 163 women. The researchers excluded 13 responses—10 without a gender response and three whose explanations were considered meaningless or uninterpretable. This was a secondary analysis and was not preregistered.

Each participant saw the same eight target texts. Seven were generated by different large language models—GPT-4o, o1, Claude 3.5, Gemini, Copilot, Llama 3.1 and Perplexity—and one was a human-written public comment. Participants rated whether each text was AI- or human-authored on a seven-point scale. The researchers later reduced those ratings to a binary classification, producing 3,120 judgements. After all eight decisions, participants gave an open-ended explanation of how they had decided.

The denominator is important. The study did not sample many human authors, genres, prompts or editing practices. It tested repeated judgements over a fixed, artificial set with seven AI texts and one human text. Although the binary scoring treated AI and human answers as two classes, the underlying set was not a broad or balanced representation of everyday writing. The results therefore describe this task, not an intrinsic human detection ability across school essays, news, workplace messages or literature.[1]

How the researchers derived four reading strategies

The team converted the open-ended Japanese explanations into sentence embeddings using a Japanese BERT model, then applied hierarchical clustering. They interpreted four clusters as unsubstantiated intuition, stylistic cues, syntactic cues and higher-order pattern recognition. The first captured answers resting mainly on a feeling; the next two referred to vocabulary, tone or sentence structure; the last described broader coherence, contextual naturalness or patterns across a whole passage.

These labels are researcher interpretations of retrospective comments, not experimentally assigned behaviours. Participants explained their approach only after completing all eight judgements, so the account may compress, rationalise or forget what happened text by text. The study also interprets the strategies along a System 1–System 2 continuum, but it did not directly measure cognitive processing, attention or response time. It cannot establish that one group genuinely used a stable mental system while another did not.

The clusters themselves were not sharply separated. The reported silhouette score was 0.1467 with two clusters, 0.0635 with three and 0.0496 with four; higher values ordinarily indicate clearer separation. Four clusters were retained partly because they were interpretable. An alternative embedding model changed cluster composition. That does not make the exercise useless, but it means the exact four-part taxonomy should be treated as an exploratory description rather than a validated scale of detection strategies.[1]

The detailed results show inconsistency, not a winning cue

Mean accuracy was 0.41 for the unsubstantiated-intuition group, 0.43 for the stylistic-cue group, 0.44 for the syntactic-cue group and 0.45 for the higher-order group. Group sizes were 61, 65, 48 and 216 respectively. Hierarchical Bayesian logistic regression adjusted for gender, age and prior LLM use, but the central practical result remained modest: the strategies did not yield a reliable separator.

Performance also varied substantially by model. Participants correctly identified 56.4% of Claude texts and 55.9% of Llama 3.1 texts, compared with 24.9% for o1 and 32.6% for Copilot. The corresponding figures were 42.3% for GPT-4o, 41.3% for Gemini and 52.3% for Perplexity. Across the seven AI texts, accuracy was 43.7%. The single human passage was correctly identified by 45.4% of participants.

Those model-specific percentages should not be turned into a league table. Each model contributed one response to one task, and generation settings, prompt wording and subsequent editing can all change detectable surface features. A different prompt or genre could reverse the pattern. Because every participant judged the same material, the study can compare reactions to those eight texts, but it cannot estimate how often a named model is detectable in the wider world.[1]

What teachers, editors and workers should do differently

For educators, the humane implication is procedural. A student's vocabulary, tidy structure or lack of idiom should not by itself become evidence of misconduct. Such cues can also reflect translation, disability, formal training, anxiety or a writer working outside their first language. Institutions should publish what assistance is allowed, ask for process evidence proportionate to the assessment and provide an appeal path. Drafts, cited sources and a short discussion of reasoning are more informative than a style-based accusation.

Editors and employers face the same risk. An automated or intuitive judgement can unfairly penalise people whose writing happens to resemble a stereotyped model output. If authorship matters, request disclosure and preserve document history; if quality matters, assess accuracy, originality, evidence and fitness for purpose directly. Moderators should separate the question 'was a model involved?' from 'does this violate a rule or cause harm?' Model assistance does not make a claim false, and human authorship does not make it true.

The study does not evaluate commercial AI detectors, watermarking or cryptographic provenance systems, so it cannot rank those tools. Its result is narrower but still consequential: unaided readers did poorly here, and naming a more elaborate reading strategy did not solve the problem. Human review remains essential, but reviewers need evidence beyond stylistic confidence.[1]

What remains unproven

The data were collected in early 2025, while the paper was published in October 2026. Models and public familiarity may have changed during that interval. The sample came from one Japanese web panel and the task was presented in Japanese, limiting transfer to other languages, countries and media environments. Prior LLM use was included as a covariate, but the design cannot determine whether training people on particular cues would improve performance or simply make them confidently wrong in new situations.

Reducing a seven-point judgement to two categories also discards confidence and ambiguity. A participant who chose the midpoint and one who expressed extreme certainty may count the same after dichotomisation. With only one human text, the study cannot describe how strategy performance varies among different human writers. Nor can it separate knowledge of a model's characteristic output from general skill at recognising authentic human context.[1]

Funding, disclosure and what would change the assessment

The work was funded by JSPS KAKENHI grant JP23K11107. The authors reported that the funder had no role in the study beyond supporting an English-editing fee and declared no commercial or financial conflict of interest. They disclosed using generative AI for English editing, followed by review by a native English speaker. Those disclosures are relevant to transparency but do not alter the evidential limits of the design.

A stronger assessment would require preregistered studies with many independently sampled human and AI texts across prompts, languages, genres and editing conditions. Researchers should preserve confidence ratings, test strategies prospectively, compare trained and untrained readers, and measure false accusations as well as successful detections. Field studies should examine fair procedures in schools and workplaces. Until then, the most defensible conclusion is practical rather than universal: in this controlled task, people and their stated strategies were not reliable enough to determine authorship on style alone.[1]