What makes healthcare AI feel fair to patients?
In a survey of 149 UK respondents, the strongest concerns were decisions that could not be questioned, systems built without clinicians and incorrect health records. The convenience sample and hypothetical scenarios measure views, not the safety or fairness of a deployed tool.
The short version
3 key points
- 1.Among 149 UK respondents, 89.9% said fairness would be harmed if patients or clinicians could not question an AI decision; 89.3% pointed to development without clinician input and 86.6% to incorrect electronic health-record data.
- 2.The questionnaire used two hypothetical vignettes and 13 structured factors, with free-text responses adding accessibility, long-term-condition complexity, maintenance and loss of empathy.
- 3.Recruitment through patient and public involvement networks produced an informed convenience sample, not a nationally representative estimate or a test of any deployed system.
The answer: people want contestable decisions, clinical input and trustworthy data
For the 149 UK respondents in this cross-sectional survey, the clearest fairness warning was an AI decision that neither a patient nor a clinician could question. That condition was judged harmful to fairness by 89.9% of participants. Development without clinician input followed at 89.3%, and incorrect information in an electronic health record at 86.6%. The pattern was similar when people considered a patient-facing symptom checker and a clinician-facing tool intended to support cancer-treatment decisions. The result places practical accountability ahead of an abstract demand that a model simply be accurate.
This is evidence about perceptions, not proof that one feature makes an algorithm objectively fair. No system made a real diagnosis or treatment recommendation, and the researchers did not compare error rates across patient groups. The study instead identifies conditions that could make adoption legitimate or unacceptable to people likely to be affected. Its most useful implication is that healthcare organisations should not treat fairness as a model metric alone: patients also care about who helped build the tool, whether the underlying record is correct, who can intervene and whether a decision can be challenged before it affects care.[1]
What the researchers asked
Participants were recruited in the United Kingdom through public and patient involvement and engagement networks. They considered two short scenarios spanning different relationships with AI. In one, a person interacted directly with a symptom checker. In the other, a clinician used an AI decision-support tool while considering cancer treatment. Respondents rated how 13 factors would affect the fairness of the resulting decision and could describe additional concerns in free text. The team reported descriptive statistics, used chi-square tests to compare attitudes, estimated associations with ordinal logistic regression and analysed the written responses thematically.
The paired vignettes were a useful design choice because they tested whether concern disappeared when a clinician sat between the system and the patient. It did not: the importance assigned to oversight, professional involvement and data quality was consistent across both settings. Free-text answers also broadened the structured list. Respondents raised accessibility, the difficulty of representing multiple long-term conditions, the need to maintain systems after launch and the possibility that automation could erode human empathy. Those themes show why predeployment consultation needs open questions rather than a checklist written solely by technical teams.[1]
What the percentages can and cannot say
The percentages are denominated by 149 survey respondents, not by the UK population. Recruitment through involvement networks may reach people who have more experience of research, health services or patient advocacy than a random household sample. Voluntary participation can also favour those with strong interest in digital health. The study is cross-sectional, so it records views at one point in time, and hypothetical choices may not predict how someone responds when ill, under time pressure or faced with a clinician's recommendation. Subgroup comparisons from a sample this size should be treated cautiously.
Nor did the survey ask whether a specific system was fair after showing audited performance, subgroup error rates or a working appeals process. Respondents were reacting to described factors, which is appropriate for eliciting priorities but cannot reveal the trade-offs they would accept in practice. A person might value human oversight and still choose an automated route if it shortened a dangerous wait; another might reject a marginally more accurate tool if there were no route to correct a record. Those decisions require service-specific evidence and deliberation, not a universal threshold inferred from this survey.[1]
What this means for patients and clinical services
For patients, contestability needs to become a visible part of the service. A health-AI interface should say what information was used, what the output means, who remains responsible and how to report an error or request human review. A clinician-facing tool needs the same protections even if the patient never sees its screen, because a recommendation can still shape access, prioritisation or treatment. Correcting an electronic record must also be possible before inaccurate data are repeatedly fed into later decisions. A nominal human in the loop is insufficient if workload or policy makes disagreement impractical.
For developers and NHS organisations, the findings support involving clinicians and patients across design, testing, procurement and post-market monitoring. That does not mean letting a small consultation group declare a system fair. Teams still need evidence of diagnostic or operational performance, subgroup analyses, accessibility testing, security review and incident reporting. They should also decide how a model will be updated when clinical guidance or patient populations change. The authors received support from Wellcome, the Daphne Jackson Trust and several National Institute for Health and Care Research programmes; they report that funders had no role in design, analysis or writing and declare no competing interests.[1]
A wider definition of fairness
Technical fairness research often asks whether error rates or benefits are distributed similarly across demographic groups. That remains essential, especially where historic care patterns have produced biased records. The survey adds a complementary layer: procedural fairness. People want a route to challenge an output, meaningful professional participation and attention to maintenance, accessibility and empathy. A system could satisfy one statistical parity measure and still feel unfair if nobody can explain who is accountable or correct an obvious mistake. Conversely, an appealing interface and clinician presence cannot excuse unequal performance.
The study is UK-specific, but the underlying governance questions travel across health systems. The practical answer will differ where clinicians are scarce, records are fragmented, or care is paid for directly. Human review may be valuable yet unavailable at the moment a remote service is used. That makes it important to test alternative safeguards rather than export a high-resource workflow unchanged. Local patient groups should help define acceptable escalation, access and explanation, while regulators and providers retain responsibility for measurable safety and discrimination risks.[1]
What would change the assessment
A nationally sampled study, deliberately including people with disabilities, limited English, low digital access and multiple long-term conditions, would show how widely these priorities are shared. Experiments could vary real design features—such as response time for human review, availability of an explanation, clinician sign-off and a record-correction route—and measure choices rather than intentions alone. Qualitative work during real deployments could then examine whether people actually use challenge mechanisms and whether those mechanisms alter care.
The strongest evidence would link procedural safeguards with audited outcomes. Researchers should measure who receives a recommendation, who contests it, how often it changes, how long review takes and whether error and access gaps narrow or widen. Patient-reported trust should be reported beside safety, effectiveness and equity, not used as a substitute for them. Until that evidence exists, the new survey provides a clear design agenda and a warning against unchallengeable automation; it does not certify any healthcare AI system as fair.[1]
What this means for people
- Patients want a practical route to question AI-influenced decisions and correct the health data behind them.
- Clinicians need enough authority, time and information to override a system rather than serve as a nominal rubber stamp.
- Health services should test accessibility and procedural safeguards alongside model performance and subgroup error rates.
Global context
The participants were recruited in the UK through patient-involvement networks. Oversight expectations, record systems, clinician availability and appeal routes differ internationally, so the priorities require local testing even when the governance questions are broadly relevant.
What the evidence does not yet show
- The 149-person convenience sample is not a nationally representative estimate of UK patient opinion.
- Responses concerned hypothetical vignettes rather than decisions made by a deployed system.
- Cross-sectional attitudes cannot establish causal effects or predict behaviour under clinical pressure.
- The study measured perceived fairness, not model performance, discrimination, safety or patient outcomes.
What to watch next
- Representative studies including digitally excluded and medically complex patients
- Real-world use and outcomes of challenge, correction and clinician-override routes
- Joint reporting of procedural fairness and subgroup performance
- Post-deployment maintenance and accountability requirements
Living evidence record
Impact record IAI-1HUXGUO
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
13 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what BMC Medical Informatics and Decision Making published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 13 October 2026
Research question, topics and regions
Which design, data and governance factors UK patients believe would make a symptom checker or cancer-treatment decision-support system more or less fair
Topics: Healthcare AI · Patient voice · Fairness · Human oversight · Clinical governance
Regions: United Kingdom · Europe
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
What do Welsh focus groups want from healthcare AI?
Across four focus groups involving 21 people, participants repeatedly asked for human contact, human oversight and secure governance. The peer-reviewed study offers useful reasons and contrasts between groups, but its small, NHS-connected sample cannot estimate what Wales as a whole believes.
7 min · 1 source
Health & Life Sciences
Can AI reliably extract breast-cancer trial data from health records?
For many fields, but not all. In a peer-reviewed single-hospital study of 113 patients in 11 breast-cancer trials, an AI pipeline matched manually curated trial forms well on several variables, yet accuracy fell to 46% for treatment start dates and 23% for end dates. The system remains an assistive extraction tool requiring human verification.
9 min · 1 source
Health & Life Sciences
What does it take to scale healthcare AI safely?
An OECD conference report turns input from 142 experts across 34 countries into ten proposed actions for health systems. It is a practical consensus framework for trust, infrastructure and financing—not evidence that scaled AI has improved patient outcomes.
10 min · 2 sources
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.