Do certified health-record AI systems explain their risks clearly?

What the researchers reviewed

The team started with the publicly available list of health-information-technology products certified under the US programme and identified products that included an AI-enabled predictive decision-support intervention. For each certified product, they downloaded the corresponding public risk-management summary, compiled the documents and coded them against the HTI-1 rule. The final denominator was 35 developers, rather than 35 algorithms or hospitals. One developer can supply multiple tools and one tool can be used in many settings, so the study cannot estimate how many patients or clinical decisions are covered by incomplete disclosures.

Two trained reviewers independently coded every summary. They recorded whether it described the risk-analysis process, named specific risks, outlined mitigation, described governance and addressed any of the eight recommended attributes. Closely related terms counted—for example, a discussion of bias mitigation could satisfy the fairness category. The reviewers also made qualitative notes about detail and resolved disagreements by consensus. The analysis is descriptive: it reports counts and patterns rather than testing a causal hypothesis or scoring clinical performance.

That design is appropriate for a disclosure audit, but it leaves some important quantities unreported in the accessible paper. The authors do not provide a formal inter-rater-reliability statistic, a product-by-product comparison or a measure of how understandable the documents were to clinicians, purchasers or patients. Accepting conceptual synonyms makes the audit less likely to miss relevant material, while also allowing a broad reference to count even when it does not disclose a test method, threshold, subgroup result or corrective action.[1]

Why the certification context matters

The HTI-1 final rule requires developers of certified health technology to make summary information about risk-management practices publicly available for predictive decision-support interventions they supply. It also requires information that helps users understand source attributes such as development, training and evaluation. The policy goal is not simply to create paperwork: a hospital choosing an embedded risk score needs to know what population it was built for, how it can fail and whether performance has been examined across groups.

The paper's sharpest finding is that every sampled product had achieved certification despite uneven public reporting. Certification can confirm that a process or documentation requirement was judged to have been met; it is not a blanket approval of every model's clinical effectiveness. The audit also does not identify which certifying body reviewed each submission or reconstruct the evidence available during review. It therefore supports closer examination of certification practice, but it does not prove that any named certifier acted improperly.

For procurement teams, the practical implication is to treat a certificate as the start of due diligence. Buyers can ask for the intended population, training and validation datasets, subgroup performance, update policy, failure reporting, local monitoring plan and a named owner for corrective action. If a summary lists 'fairness' or 'security' without a method and result, the buyer still lacks the information needed to judge the claim.[1][2]

What this means for clinicians and patients

Clinicians often encounter decision support inside the record rather than as a separate AI product. A prediction may appear as a score, warning or recommended action during an already busy workflow. If the system's limits are difficult to find, users can over-trust a result, dismiss alerts indiscriminately or be unable to explain why the tool behaved differently for a particular patient. Clear public summaries do not solve those problems, but they make independent scrutiny and local governance more possible.

Patients may never know that an embedded model influenced triage, diagnosis or treatment planning. The study did not examine patient notices, consent, appeal routes or outcomes, so it cannot say whether anyone was harmed. It does show that the public layer intended to support accountability is inconsistent. A usable standard should make it possible to learn what the model is for, who is accountable, which groups were evaluated, what important failures are known and how a concern can be raised without requiring a reader to decode vendor-specific terminology.

Smaller health systems face a particular burden. Large academic centres may employ informaticians, safety scientists and lawyers to interrogate a vendor. A community clinic may rely more heavily on certification and concise public evidence. Standardised, comparable reporting could therefore reduce an information disadvantage, provided that the template demands concrete methods and results rather than another set of boxes.[1][2]

The audit cannot tell us whether the AI is safe

The researchers examined public documents, not source code, internal risk registers, incident logs or deployed performance. A developer with a thin summary may have extensive internal controls; a developer with a polished summary may still operate a poorly performing model. The study does not compare disclosures with real practices, independently verify any mitigation or test whether a product improves care. It also captures an early period after the reporting requirement took effect, when developers and certifiers may still have been interpreting the rule.

The denominator is small and confined to one national certification system. The findings cannot be transferred directly to medical devices regulated through other pathways or to health systems in Europe, Asia, Africa or Latin America. They are nonetheless relevant beyond the United States because many multinational vendors sell into several jurisdictions, and regulators everywhere face the same distinction between requiring a disclosure and verifying that it is complete, comparable and useful.

The authors report no external funding and declare no competing interests. The article is peer reviewed, although the journal labels the available page as an accepted early-view version that may still receive copyediting. The work was first circulated as a preprint in December 2025; the portal uses the peer-reviewed journal publication date of 14 October 2026 and does not present the older preprint as new evidence.[1]

What evidence would change the assessment

A stronger follow-up would publish the coding instrument, anonymised document-level scores and inter-rater agreement, then repeat the audit after developers have had more time to comply. Usability testing should ask clinicians, procurement staff and patient representatives to find and interpret essential facts. Regulators could also compare disclosures with confidential audit material, corrective actions and post-market incidents to learn whether better reporting predicts better risk management.

The most consequential test would connect documentation to deployed performance. For a defined set of algorithms, independent evaluators could compare the public claims with local validation, subgroup error rates, alert overrides, safety events and update histories across multiple hospitals. Evidence that standardised reporting improves procurement, catches unsafe systems earlier or leads to faster correction would support a stronger claim than this audit can make. Until then, the defensible conclusion is precise: US rules have created public summaries, but certification has not yet made those summaries consistently detailed or comparable.[1][2]