Did ChatGPT teach consultants data science? It lifted task scores, but the knowledge did not stick
In a randomized experiment, GPT-4 helped generalist consultants approach expert performance on three technical tasks. A test immediately afterwards found no clear gain in unaided technical knowledge.
The short version
- What happened
- A randomized experiment analysed 487 BCG consultants after 986 were invited; participants completed two of three 90-minute data-science tasks, with 44 BCG data scientists providing an expert benchmark.
- Who it affects
- Workers may complete unfamiliar technical tasks more effectively with AI, but employers should not relabel temporary tool-assisted output as a learned professional skill.
- What remains unproven
- 487-person analysis sample drawn from one highly educated consulting workforce
Keep in mind
487-person analysis sample drawn from one highly educated consulting workforce
Short tasks and an immediate post-test cannot establish durable learning
No live client work, job-transition outcome or end-to-end data-science assessment
The study ran in March and April 2024 using GPT-4-era tooling
BCG and OpenAI affiliations and direct study involvement require consideration
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
GPT-4 closed much of the task-performance gap
A peer-reviewed randomized experiment found that GPT-4 acted as a strong temporary performance aid for generalist consultants tackling work normally outside their expertise. Boston Consulting Group invited 986 consultants; 573 started and 487 formed the analysis sample. Each completed two of three 90-minute tasks—coding, statistical understanding and prediction—while 44 BCG data scientists supplied an expert benchmark.
The treatment group received 20 minutes of training plus ChatGPT; the control group received matched training and could use public online resources. ChatGPT access increased coding scores by 49 percentage points versus control (95% confidence interval 42–56), statistics scores by 20 points (15–25), and prediction similarity to expert answers by 17 points (9–25). Treated consultants still remained below the data-scientist average on coding and statistics.[1]
Better output was not the same as learning
The important limit came immediately afterwards: researchers found no evidence that treated participants had gained more technical knowledge when answering without ChatGPT. In an exploratory measure, they were also worse at judging which problems the system could solve. For workers, this separates assisted performance from a durable new skill; for employers, it cautions against treating higher short-term output as proof that training has succeeded.
The participants were highly educated consultants from one firm, the tasks were brief, and the experiment ran in March and April 2024. It did not test delayed retention, messy end-to-end projects, job mobility or whether errors would be caught in live work. Authors include BCG and OpenAI researchers, and the two organizations helped supply participants, benchmarks or task design. Independent replication with diverse workers and delayed unaided tests would change the assessment.[1]
The distinction that matters
Where ChatGPT changed performance
Effects compare GPT-4 access with matched training plus ordinary public online resources; they are not gains over doing nothing.
Coding
Consultants completed a 90-minute technical task
+49 percentage points (95% CI 42–56)
Statistics
Consultants interpreted statistical evidence
+20 points (95% CI 15–25)
Prediction
Answers were compared with expert response vectors
+17 points closer (95% CI 9–25)
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 11 min 27 sec · Background explainer
UNESCO Talks: Interview with 2024 Nobel laureate Professor Daron Acemoglu
YouTube loads only when you choose; playback starts when you press play.
Economist Daron Acemoglu discusses how choices about AI affect workers, inequality and society. Pair the interview with our comparison of task exposure, hiring and skills evidence.
Evidence check: An expert interview from 2024, offering a perspective on policy and work rather than a measured forecast of job losses.
Published 2024-10-14. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the AI and jobs guidePhoto: Dylan Gillis / Unsplash · Licence
Continue the story
Read next
Continue exploring knowledge work: compare the finding and its evidence limits.
A slower start helped people write better with AI—and the benefit carried into the next task
A Microsoft Research preprint randomised 297 US workers to an AI writing interface with or without planning prompts. Quality gains were small, and the study did not test long-term workplace performance.
Read the next reportExplore the full source and evidence record
What this means for people
- Workers may complete unfamiliar technical tasks more effectively with AI, but employers should not relabel temporary tool-assisted output as a learned professional skill.
Global context
The experiment was conducted inside BCG, but the distinction between assisted performance and retained skill matters internationally wherever employers are redesigning roles or training around generative AI. Transfer to other occupations, languages and levels of digital access is untested.
What the evidence does not yet show
- 487-person analysis sample drawn from one highly educated consulting workforce
- Short tasks and an immediate post-test cannot establish durable learning
- No live client work, job-transition outcome or end-to-end data-science assessment
- The study ran in March and April 2024 using GPT-4-era tooling
- BCG and OpenAI affiliations and direct study involvement require consideration
What to watch next
- Delayed unaided tests weeks or months after AI-supported work
- Replication across less-selected workforces, countries and occupations
- Performance on end-to-end projects with real quality and accountability costs
- Whether workers learn to recognise the model's capability boundaries
Living evidence record
Impact record IAI-1SFGIL5
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
26 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Nature Human Behaviour published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 26 October 2026
Research question, topics and regions
Whether access to GPT-4 temporarily enables generalist consultants to perform data-science tasks and produces transferable technical knowledge
Topics: Knowledge work · Worker training · Data science · Generative AI
Regions: North America · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.