What are clinicians asking an AI evidence platform?
A JAMA study counted 271 million OpenEvidence queries from 952,008 US clinicians. Treatment and management dominated a classified sample—but the logs cannot show whether answers were correct, used in care or improved patient outcomes.
The short version
3 key points
- 1.From January 2024 through June 2026, 952,008 verified US clinicians submitted 271 million queries to OpenEvidence; 556,200 users were physicians.
- 2.In an LLM-classified stratified sample of 38.8 million queries, treatment and management represented 33.1% of physicians' queries, while diagnosis represented 13.9%.
- 3.The observational logs measured questions, not answer quality, clinical actions or patient outcomes. OpenEvidence provided the data and three authors disclosed company employment or leadership roles.
The answer: clinicians mainly asked about treatment, but use is not proof of benefit
Clinicians using one large AI evidence platform asked more often about treatment and management than diagnosis. A peer-reviewed JAMA study analysed 271 million queries submitted to OpenEvidence by 952,008 verified US clinicians between 1 January 2024 and 30 June 2026. Among physicians, treatment and management accounted for 33.1% of an LLM-classified query sample, medication information for 14.1% and diagnosis for 13.9%. The pattern matters because many clinical-AI evaluations still emphasise examination questions or diagnostic vignettes rather than the treatment and evidence questions clinicians appear to bring to the tool.
The study does not show that an answer was correct, followed or beneficial. It could not tell whether a query was entered during a consultation, while preparing for care or for general learning. It did not link questions to prescriptions, referrals, errors, time saved, specialist consultations or patient outcomes. The safe interpretation is therefore behavioural: this platform became a widely used route to clinical information, and its users frequently sought guidance on consequential treatment decisions. That raises the priority of evaluating those answers in realistic workflows; it does not establish that the platform improved care.[1]
How the researchers counted clinicians and queries
The cross-sectional study used OpenEvidence query logs for accounts with a National Provider Identifier, then matched those identifiers to the US National Plan and Provider Enumeration System. Accounts without a matching record and clinicians who submitted no queries were excluded. Users were categorised as physicians, nurse practitioners, physician assistants, pharmacists or other licensed clinicians. A query was one free-text question that generated a synthesised response; each follow-up question counted separately. Institution-specific deployments and product features outside the main question interface were not included.
Across the 30-month period, 952,008 clinicians submitted 271 million queries. Physicians represented 556,200 users and 172.1 million queries. The remaining users included 171,034 nurse practitioners, 93,241 physician assistants, 35,819 pharmacists and 95,714 other licensed clinicians. Monthly active users—defined as anyone submitting at least one query in that month—rose from 355 in January 2024 to 703,265 in June 2026. Monthly query volume rose from roughly 6,000 to 27.8 million, while average monthly queries per active user increased from 18.0 to 39.5.[1]
A 38.8-million-query sample supplied the purpose labels
The authors did not manually read all 271 million questions. They used a large language model to assign the primary purpose and clinical topic of a stratified random sample of 38.8 million queries. Purpose categories included clinical decision-making, clinical knowledge, evidence, administrative tasks and medical education. Reported percentages were adjusted using logistic regression with marginal standardisation over sex, specialty and years in practice, except when a characteristic defined the comparison or did not apply across clinician groups.
Among physicians, 51.9% of classified queries concerned clinical decision-making, 37.2% clinical knowledge, 7.0% evidence, 3.0% administrative tasks and 0.9% medical education. Treatment and management was the largest subcategory at 33.1%, followed by medication information at 14.1% and diagnosis at 13.9%. Pharmacists had a different pattern: medication information represented 32.8% and treatment and management 32.0%. The classifier underwent blinded validation and agreed with clinician annotators at least as often as clinicians agreed with one another, but the paper still treats the labels—especially specialty topics—as useful heuristics rather than definitive human coding.[1]
Use differed by profession, career stage and specialty
Nurse practitioners and physician assistants each submitted an average of 44.5 queries per active user-month, compared with 37.1 for physicians. Among physicians, residents and fellows submitted 1.84 times the share of queries expected from their share of the workforce. Physicians within ten years of graduation had a use ratio of 2.54. Oncology and haematology, internal medicine and family medicine also contributed larger query shares than their workforce representation, with ratios of 2.00, 1.88 and 1.80 respectively.
Those ratios are not adoption rates. They compare a subgroup's share of physician queries with its share of the national physician workforce, and reliable workforce denominators were available only for physicians. A ratio above one can reflect more users, more intensive use or both. Lower ratios in ophthalmology, radiology and orthopaedics may also reflect the limits of a text-based evidence tool for imaging, procedures and other specialty workflows rather than lower interest in AI generally.[1]
Doctors often asked beyond their own specialty
The study found substantial cross-specialty querying. Thirty-five per cent of questions from oncologists and haematologists addressed another specialty, rising to 70% for surgeons, 73% for anaesthesiologists and 88% for emergency physicians. Questions within a physician's specialty were more often about evidence, literature and guidelines, while questions outside it were more often about disease and medication information. That may reflect the reality that clinicians must make decisions across disciplinary boundaries, particularly in emergency, generalist and complex-care settings.
It does not show whether an AI query replaced a colleague, preceded a referral or simply supplemented a conventional source. The logs cannot distinguish uncertainty resolved safely from false reassurance. Health services should therefore avoid reading the cross-specialty figures as evidence that consultation can be automated. A safer operational question is where an AI answer might influence care outside the user's deepest expertise and what escalation, citation checking or specialist review should be required there.[1]
What clinicians and health systems should require
For clinicians, the results make treatment and medication questions the priority for scrutiny. A useful interface should expose the underlying guideline or study, its date, population and strength, and make uncertainty visible before a recommendation is acted upon. Local policy should be explicit about entering patient information, documenting consequential use and escalating when evidence is conflicting, outside scope or not applicable to the individual in front of the clinician. A cited answer still needs clinical judgement; citation presence does not prove that the source supports the generated conclusion.
For clinical educators and supervisors, the higher relative use among trainees and early-career doctors argues for observed practice rather than a blanket prohibition. Training can require users to state the question, inspect the source, identify missing patient factors and explain when they would not follow the output. For public-service leaders, procurement and governance should test the questions staff actually ask—treatment selection, dosing, interactions, evidence synthesis and cross-specialty queries—rather than relying on diagnostic benchmark scores that may not represent routine use.[1]
Commercial involvement and the limits of a single platform
OpenEvidence supplied the data and contributed to the analysis. Three authors disclosed direct company roles: Samuel Finlayson and Mondira Ray were employees, and Travis Zack was chief medical officer. Other authors held academic and clinical affiliations, and the paper reports Harvard Medical School institutional review-board approval with a waiver of consent for secondary analysis. Those disclosures do not invalidate the study, but they increase the importance of independent replication, auditable sampling and a clear separation between describing adoption and demonstrating clinical value.
The dataset represents one US platform with access restricted to professionals holding a valid National Provider Identifier. OpenEvidence also has content partnerships with major medical publishers, so its use pattern may not transfer to tools with different evidence access, interfaces, costs or user verification. Counts from the main interface exclude other product features and institutional deployments, making them lower bounds for this platform while still providing no measure of total clinical-AI use. The study is large, but size cannot remove selection, classification and outcome gaps.[1]
What would change the assessment
Confidence would rise with independent studies across several platforms, countries and health systems using comparable definitions of active use. Human audit samples should report agreement separately for high-risk categories such as medication dosing and treatment selection. Research should connect queries to de-identified response quality, source fidelity and whether clinicians accepted, modified or rejected the output, while preserving patient privacy and avoiding surveillance of individual professionals.
The decisive evidence is prospective and patient-centred: when clinicians use an AI evidence platform for the tasks documented here, do decisions become more accurate, timely and equitable without increasing medication errors, inappropriate confidence or unnecessary referrals? Randomised or carefully controlled workflow studies, followed by monitoring for subgroup harms and evidence drift, would move the assessment from adoption to benefit. Until then, the JAMA paper establishes what many users ask one platform—not whether the platform answered well enough to improve care.[1]
What this means for people
- Patients may be affected when clinicians use AI answers for treatment, medication or cross-specialty questions, but this study did not measure the decisions or outcomes that followed.
- Clinicians need rapid access to the underlying source, its date and its applicable population before acting on a generated synthesis.
- Clinical supervisors and health-service leaders should evaluate the treatment and medication tasks staff actually bring to AI, not only diagnostic benchmark performance.
Global context
The study covers one US platform whose professional accounts were linked to National Provider Identifiers. Its scale is unusual, but the findings cannot be assumed to describe clinicians in countries with different licensing systems, evidence access, languages, health-service organisation or product availability.
What the evidence does not yet show
- Observational logs from one commercially involved platform rather than a representative sample of all clinicians or AI tools
- Queries were counted, but response accuracy, clinical actions, time savings, referrals and patient outcomes were not measured
- A large language model classified a 38.8-million-query sample, creating residual misclassification despite clinician validation
- The data could not distinguish point-of-care use from preparation, education or general information seeking
- Reliable national workforce denominators were available only for physicians, and platform employees participated in the study
What to watch next
- Independent replication across multiple clinical information platforms and countries
- Prospective evaluation of treatment, medication and cross-specialty answers against current evidence
- Links between platform use, clinician decisions, specialist consultation and patient outcomes
- Audits of citation fidelity, subgroup performance and harmful confident answers
- Governance that protects patient information while allowing meaningful safety monitoring
Living evidence record
Impact record IAI-0EDR4WM
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
15 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what JAMA published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 15 October 2026
Research question, topics and regions
How verified US clinicians used one AI-enabled clinical information platform, including query volume, purpose, specialty and career stage
Topics: Clinical AI · Evidence synthesis · Clinician decision support · Treatment · Workforce · Patient safety
Regions: United States · North America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Did clinicians prefer AI discharge summaries after long hospital stays?
In a retrospective 60-case comparison, 12 physicians usually preferred GPT-5.2 summaries and annotated fewer omissions. Reviewers knew which summary was AI-written, one hospital supplied the records, and no patient outcome or time saving was tested.
7 min · 3 sources
Health & Life Sciences
Can a high-AUC diabetes model still be unsafe?
New analysis today of a peer-reviewed 2 October audit of 12 machine-learning approaches on two public diabetes datasets. Similarly ranked models can differ materially in calibration, uncertainty and safe deferral, but this is a benchmark study—not a clinical trial, diagnostic approval or evidence of improved patient outcomes.
7 min · 3 sources
Health & Life Sciences
Can an AI agent safely change a clinical record?
A 43-task benchmark found that long-term memory helped a GPT-4.1-mini agent use FHIR tools, but the strongest held-out result was only 60.6% task success. This was a resettable test server—not a live hospital record system.
8 min · 3 sources
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.