Can AI make surveys more precise without replacing people? An 83-outcome benchmark found modest gains
An unreviewed US-data study paired real survey responses with millions of census-scale records and corrected AI predictions against people. Supervised models added more precision than LLMs, while every valid estimate still depended on human responses.
The short version
- What happened
- The unreviewed benchmark covered 37 General Social Survey outcomes and 46 National Health Interview Survey outcomes, paired with American Community Survey files containing 3.19 million and 3.37 million person records.
- Who it affects
- Survey respondents remain essential because their answers estimate and correct model error.
- What remains unproven
- The paper is an unreviewed methods benchmark, not a live test of lower fieldwork cost or better public decisions.
Keep in mind
The paper is an unreviewed methods benchmark, not a live test of lower fieldwork cost or better public decisions.
It uses US GSS, NHIS and ACS data with harmonized covariates; results may not transfer to other countries or survey frames.
Population mismatch, nonresponse, weak predictions and small subgroup samples can remove gains or invalidate inference.
Average effective-sample-size gains do not apply equally to every outcome, subgroup or regression coefficient.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer: predictions can add precision only when people correct them
A new unreviewed methods paper finds that artificial-intelligence predictions can modestly improve the precision of some US survey estimates without treating synthetic answers as people. The key is prediction-powered inference: a model predicts an outcome across a very large auxiliary dataset, while a probability survey of real respondents measures the model's errors. The final estimate combines the broad prediction with a human residual correction. If the model is systematically high or low, the observed survey pulls the estimate back. The method therefore uses prediction as auxiliary information rather than as a substitute population.
Across 83 outcomes, conventional supervised models delivered larger average gains than language models. For means in the National Health Interview Survey, the historical learner reduced confidence-interval width by 5.4% and produced the precision of a survey with 12.2% more observations. In the General Social Survey, the corresponding figures were 3.4% and 7.5%. The best LLM results raised effective sample size by 2.6% in NHIS and 2.8% in GSS. Those gains are real within the benchmark but much smaller than claims that synthetic respondents could eliminate fieldwork.
The authors are explicit that replacement is unsafe. Language models can produce biased, under-dispersed or demographically distorted answers that look precise when generated at scale. Prediction-powered correction works only because humans remain the source of validity. The study does not show that every survey will become cheaper, nor that AI can repair nonresponse, bad questions or missing groups. It evaluates statistical efficiency under specific alignment conditions using public US datasets and models available for the experiment.[1]
The denominator: 83 outcomes, two surveys and two very large auxiliary files
The benchmark used 37 political, social and wellbeing outcomes from the 2022 General Social Survey, whose public target wave contained 3,544 respondents. Historical GSS waves from 1988 to 2021 supplied 46,971 observations for training and prompt examples. It also used 46 health behaviour, health status and mental-health outcomes from the 2017 National Health Interview Survey. The processed target wave contained 78,132 observations, and earlier NHIS waves supplied more than six million historical observations. Unordered outcomes with more than two categories were excluded.
The auxiliary files came from American Community Survey public-use microdata harmonized by IPUMS USA: roughly 3.19 million person records for 2017 and 3.37 million for 2022. The researchers aligned age and population restrictions as closely as possible and retained design weights, strata and primary sampling units where available. Shared predictors included sex, age, marital status, race, nativity, citizenship, education, employment, occupation, industry, income and poverty. The ACS is census-scale, not a complete census, and it does not contain the focal survey outcomes.
Three prediction regimes were compared. A Super Learner ensemble combined generalized linear models, elastic net and random forests trained on earlier survey waves. A cross-fitted learner trained on part of the target wave and predicted held-out respondents, then scored the auxiliary file with a full-wave model. LLMs received a structured demographic profile, the verbatim question and allowed response options. Prompted systems included GPT-5-mini, Gemini 2.5 Flash and Qwen3, with one Qwen3 variant fine-tuned on historical responses. Model outputs were mapped to the same numeric codes as human answers.[1]
The distinction that matters
How prediction-powered survey augmentation works
Predictions supply breadth; sampled people supply the correction and the claim to population inference.
Predict
Score a large auxiliary file
A broad but potentially biased mean
Observe
Collect real survey answers
Human outcomes and prediction errors
Correct
Add the weighted residual
The survey target is restored
Compare
Estimate design-based uncertainty
Keep only genuine precision gains
Why the gains varied across questions and estimands
Prediction helps when the error left for the survey to estimate varies less than the original outcome. In NHIS, the largest mean effective-sample-size gains were about 34% to 36% for self-rated health, functional limitations and minutes of vigorous physical activity. Smoking frequency and some mental-health intensity measures were harder to predict from the harmonized covariates. In GSS, the largest mean gains were about 21% to 23% for computer use, marijuana legalization, home ownership and social class. Many political and moral attitudes showed much smaller gains.
The average differs by training data as well as subject matter. NHIS supplied millions of historical records, while GSS supplied tens of thousands. The study therefore cannot prove that health is intrinsically more predictable than opinion. Richer inputs could change the ranking, but they could also increase privacy and fairness risks. For self-rated health, a cross-fitted supervised model reached a prediction-response correlation of about 0.33 with 100 training observations, 0.46 with 1,000 and 0.49 with 5,000. GPT-5-mini was about 0.443 and Gemini 2.5 Flash about 0.422, making LLMs potentially useful before many labels exist.
Outcome means benefited more consistently than regression coefficients. The authors estimated adjusted associations for sex, age, education and race, and those gains were smaller and more heterogeneous. A model can predict an outcome reasonably well without reducing uncertainty in the particular score contribution that defines an adjusted coefficient. Researchers therefore cannot cite an overall R-squared or mean gain as proof that a subgroup difference is more precise. They must calculate the improvement for the exact estimand and inspect performance within the groups being compared.[1]
The statistical guardrails are the practical story
The correction requires the labelled survey and auxiliary file to represent the same target population after weighting and restrictions, and their covariates must be measured compatibly. If coverage differs, the prediction terms may not cancel. Prediction-powered inference does not fix survey nonresponse or coverage bias that the weighting fails to address. Weak or badly scaled predictions can leave residual variance unchanged or make it larger. Very small subgroups or few independent sampling units can also make the correction unstable and large-sample confidence intervals unreliable.
A simulation in the paper illustrates why precision should not be confused with truth. With 1,000 survey cases, predictive R-squared of 0.30 and 10,000 auxiliary records, effective sample size rose to about 1,370. At R-squared 0.60 it rose to about 2,174, or 2,463 with 100,000 auxiliary records. Because p-values respond nonlinearly to precision, the same assumed effect moved from p≈0.114 in the survey-only calculation to about 0.020 or 0.013 in the stronger scenarios. The prediction did not create a larger effect; it changed estimated uncertainty under the stated assumptions.
For survey agencies, the promising use is selective: publish harmonized auxiliary links, compare ordinary and augmented estimates, and reserve interview time for outcomes and groups where prediction is weak. For the public, the safeguard is simpler: an AI-generated answer should never be counted as a person without a valid design and observed correction. The work is an unreviewed US benchmark funded by Hong Kong's Research Grants Council. Replication across countries, survey frames, rare groups and live fieldwork would change confidence. The method could increase the return on each human interview, but only by preserving—not erasing—the interview.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring public opinion: compare the finding and its evidence limits.
Why did AI become experts’ top future risk?
In AXA’s 2026 perception survey, AI and big data rose from fourth to first among 4,345 risk professionals, while 24,000 adults ranked it second behind climate change. The result measures concern, not the probability or cost of future harm—and 89% of the expert panel worked for AXA.
Read the next reportExplore the full source and evidence record
What this means for people
- Survey respondents remain essential because their answers estimate and correct model error.
- Public agencies could spend scarce interviews on less predictable questions or underrepresented groups, but only after subgroup validation.
- Researchers must report survey-only estimates alongside augmented results so prediction does not hide model dependence.
Global context
The United States has unusually large public-use ACS microdata. Other countries may rely on registries or smaller auxiliary frames with different access, privacy and legal conditions. Any transfer needs local population alignment, weights and governance.
What the evidence does not yet show
- The paper is an unreviewed methods benchmark, not a live test of lower fieldwork cost or better public decisions.
- It uses US GSS, NHIS and ACS data with harmonized covariates; results may not transfer to other countries or survey frames.
- Population mismatch, nonresponse, weak predictions and small subgroup samples can remove gains or invalidate inference.
- Average effective-sample-size gains do not apply equally to every outcome, subgroup or regression coefficient.
What to watch next
- Peer review and independent replication.
- Prospective fieldwork tests with preregistered stopping rules.
- Subgroup-specific precision and fairness audits.
- Comparisons with non-AI calibration and imputation baselines.
Living evidence record
Impact record IAI-1FRWS80
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether externally generated predictions can reduce the sampling uncertainty of probability surveys while real respondents supply a design-based correction
Topics: Surveys · Public opinion · Prediction-powered inference · Synthetic data
Regions: Hong Kong · United States
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.