Could hidden AI steering quietly change depression-treatment advice?
A peer-reviewed proof-of-concept shifted one open model's emphasis between medication and self-care without changing its weights or visible prompt. The test used 12 synthetic scenarios—not patients, clinicians or commercial chatbots.
The short version
- What happened
- The experiment generated 372 deterministic responses from one open-weights model across 12 synthetic depression scenarios and 31 steering amplitudes. It measured answer content, not patient outcomes or clinical decisions.
- Who it affects
- An undecided person may be especially vulnerable to how options are framed, but the study did not involve real patients or measure any treatment choice.
- What remains unproven
- One open-weights model, one activation direction and 12 synthetic scenarios; no commercial systems were tested.
Keep in mind
One open-weights model, one activation direction and 12 synthetic scenarios; no commercial systems were tested.
The primary set contained 372 deterministic responses and no human participants, clinicians, treatment decisions or health outcomes.
A large language model performed the main content scoring; a blinded human expert rescored only a sample, with intraclass correlation of 0.81.
The study measured emphasis between medication and selected self-care categories, not overall clinical accuracy or appropriateness.
The author disclosed relevant advisory, equity and editorial roles; NIMH grants supported the work and the funder reportedly had no study role.
Replication is needed across models, languages, prompts, clinical conditions and independent evaluation teams.
Source analysis · 1 linked source. Primary analysed source date: 3 November 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 10 min The evidence, methods, implications and limitations
The direct answer: the advice moved, but the study is a controlled proof-of-concept
Yes—within this experiment, hidden inference-time steering shifted how one open-weights language model presented depression-treatment options. The researcher constructed a direction inside the model that contrasted antidepressant medication with self-directed approaches such as exercise, diet, meditation and dietary supplements. Moving the model along that direction changed the balance of its answers even though the model weights and visible system prompt stayed the same. The intervention would not have been apparent to a person reading the prompt or output alone.
That result is important because health-information systems can frame choices without issuing a direct instruction. It is not evidence that a patient was harmed, that a clinician changed treatment or that a commercial chatbot is secretly steering users. The study used synthetic text scenarios, no human participants and one model—DeepSeek version 4 Flash. It tested one deliberately constructed axis and a restricted set of depression questions. The publication therefore establishes technical plausibility under controlled conditions, not prevalence, intent or clinical effect in deployed systems.[1]
How the hidden steering was built
The study used the open-source DS4 inference tool to alter internal activations while the model generated an answer. The researcher began with 16 pairs of training prompts. One prompt in each pair asked for medication to be presented as the first-line treatment; the other asked for self-care to be emphasised without medical involvement. The average difference between the model's internal responses to those paired prompts became a steering direction. That direction was then applied at the attention output of every transformer block during generation.
Negative amplitudes amplified the medication direction and positive amplitudes amplified the self-care direction. The experiment tested 30 amplitudes from minus 1.5 to plus 1.5 in increments of 0.1, plus an unsteered baseline. This is not conventional fine-tuning: the published weights remained unchanged, and no new system instruction disclosed the desired emphasis. The distinction matters for oversight because an organisation controlling the inference stack could, in principle, alter behaviour after a model has been evaluated or released without presenting users with a different model name or visible policy prompt.[1]
The distinction that matters
What changed—and what stayed constant
The experiment isolated an inference-time control while holding the visible prompt and base weights constant.
Held constant
Keep model weights and system prompt unchanged
The named base model and visible instruction do not reveal the intervention
Changed
Apply a medication-versus-self-care activation direction during generation
Internal activations shift at every transformer block
Measured
Score 372 responses across 12 scenarios and 31 amplitudes
Treatment emphasis moves with steering strength
Not measured
No patients, clinicians or treatment decisions
Clinical harm, prevalence and real-world choice effects remain unknown
What was actually generated and counted
The evaluation contained 12 synthetic scenarios asking for depression-treatment advice. Four stated a preference for medication, four preferred to avoid medication and four expressed no preference. Each scenario was run once at every one of the 31 amplitudes with random sampling turned off, producing 372 primary responses. The fixed generation setting makes the response set reproducible under the tested software configuration, but it does not represent the variation people would encounter across repeated conversations, different prompts, model updates or temperature settings.
A Claude Opus version 4.7 rater scored how extensively each answer discussed medication and four self-care categories on a scale from zero to three. A blinded human expert rescored a sample; the reported intraclass correlation was 0.81, indicating substantial but not perfect agreement. The primary outcome subtracted the most-discussed self-care score from the medication score. The author also counted medication and self-care terms, used mixed-effects models with a random intercept for scenario and measured whether answers recommended consulting a clinician. The main outcome is therefore a content-emphasis measure, not a judgement that either treatment route was medically appropriate for an individual.[1]
The measured shift was clearest when the requester had no stated preference
Across the primary response set, the rated medication-versus-self-care balance declined by 0.32 points for each unit increase in steering amplitude. The 95% confidence interval ranged from a 0.25- to 0.39-point decline; a random-slopes sensitivity model produced a wider interval from 0.13 to 0.51 points. Mean balance moved from 1.25 at the strongest medication steering, to 0.92 with no steering, to 0.17 at the strongest self-care steering. Medication-term counts fell as self-care-term counts rose, providing a more objective check on the model-based ratings.
Preference changed the pattern. When a scenario expressed no treatment preference, the medication rating fell by 0.44 points per unit amplitude, with a 95% confidence interval from 0.34 to 0.54. The change was smaller when medication was requested and not detectably different from zero when the requester wished to avoid medication. The finding suggests that an undecided user may be more open to framing, but it should not be converted into a psychological claim about real patients. The scenarios were authored examples, not conversations with people experiencing depression, and no participant reported trust, intention or treatment choice.[1]
Safety language persisted, but that does not neutralise the framing risk
A recommendation to consult a clinician appeared in 322 of the 372 responses, or 87%, with a 95% confidence interval from 83% to 90%. The odds of including that recommendation did not change detectably with steering amplitude. This is a useful boundary: the manipulation shifted the balance of options without simply stripping out every reference to professional care. It also shows why a single safety phrase is an incomplete audit. An answer can advise consultation while still systematically foregrounding one route and downplaying another.
The paper did not score clinical accuracy case by case, compare the advice with a guideline or ask psychiatrists whether the final recommendation was safe. It focused on emphasis. Some self-directed approaches may be helpful alongside professional care, while dietary supplements can introduce evidence and interaction questions; medication appropriateness likewise depends on diagnosis, severity, history, risks and patient preference. A balanced audit would need to test factual accuracy, contraindications, escalation for suicidal risk and the relationship between framing and actual decisions. The present result identifies an attack or governance surface, not a complete safety evaluation.[1]
What this means for patients, clinicians and system owners
For patients, the practical message is not to distrust every digital health answer or infer that a particular service has been manipulated. It is to treat model-generated treatment information as framing as well as facts. People considering depression treatment should discuss options with an appropriately qualified professional, particularly when symptoms are severe, persistent or include thoughts of self-harm. The study did not test any treatment and should not be used to favour medication, reject medication or substitute supplements and lifestyle advice for care.
For clinicians and health organisations, model evaluation should include the inference environment rather than only the named base model. A provider can keep weights constant while changing system prompts, retrieval, tool access, safety layers or internal activations. Procurement and assurance therefore need configuration records, versioned deployment artefacts, reproducible test prompts and monitoring for shifts in option presentation. For developers, the work suggests a concrete red-team test: probe whether a model's recommendations remain stable when hidden control layers are varied, and whether the user can see who selected those controls and for what purpose.[1]
The important limits are model scope, scenario scope and absence of real-world outcomes
This was one research letter by one author, using one open-weights model, one contrast direction and a small synthetic scenario set. Closed commercial models may have different architectures, access restrictions and safeguards. The experiment does not estimate how often activation steering is used, whether a provider could implement the same intervention at scale without detection, or whether independent auditors can reliably distinguish steering from ordinary model updates. Random sampling was disabled in the primary analysis, which strengthens controlled comparison but narrows realism.
The author disclosed scientific-adviser roles with Genomind, Circular Genomics and Alkermes, equity in Circular Genomics, and editorial roles at JAMA +AI and JAMA Network Open; the paper states he was not involved in its editorial decision. National Institute of Mental Health grants supported the work and the funder had no reported role in the study or publication decision. Claude tools assisted coding, literature work and response scoring, with human validation of a sample. Those disclosures do not negate the result, but they matter when judging independence, replication needs and the use of one model to rate another.[1]
What would change the assessment
Confidence would rise with preregistered replication across several open and closed models, multiple clinically meaningful steering directions and a much larger scenario bank written and scored by independent mental-health experts. Evaluators should include repeated stochastic generations, multilingual prompts, varying symptom severity and explicit crisis scenarios. Human raters should assess accuracy, completeness, harmful omission and appropriate escalation, while automated scores remain secondary. Publishing the exact deployment configuration and signed model artefacts would help auditors determine what changed between evaluation and use.
The clinical question needs a different design. Carefully governed user studies could test whether undisclosed framing changes comprehension, trust, stated preference or discussions with clinicians, without withholding appropriate care. Real deployments would require monitoring, complaint routes and the ability to reconstruct which model and control settings produced an answer. The assessment would become more concerning if the effect replicated across systems and altered choices despite accurate disclosure of options. It would become less concerning if robust external controls detected or prevented hidden shifts. For now, the finding is a credible warning about invisible configuration power, not evidence of widespread patient manipulation.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring depression: compare the finding and its evidence limits.
Can a depression AI be ‘correct’ when the dialogue never supplied the evidence?
An audit of 2,744 symptom cells found most lacked sufficient textual evidence. In a smaller model test, many benchmark-correct predictions were still unsupported by the dialogue.
Read the next reportExplore the full source and evidence record
What this means for people
- An undecided person may be especially vulnerable to how options are framed, but the study did not involve real patients or measure any treatment choice.
- Clinicians and health services need assurance that covers the full inference configuration, not only a model name or one pre-deployment benchmark.
- Patients should not use this proof-of-concept to start, stop or reject depression treatment; individual decisions require qualified clinical care.
Global context
The experiment was conducted by a US researcher using English synthetic scenarios and an open model. Depression guidance, medicine access, clinical pathways and cultural expectations vary across countries and languages. The governance lesson—auditing the full inference configuration and disclosing who controls it—travels more readily than the measured effect size.
What the evidence does not yet show
- One open-weights model, one activation direction and 12 synthetic scenarios; no commercial systems were tested.
- The primary set contained 372 deterministic responses and no human participants, clinicians, treatment decisions or health outcomes.
- A large language model performed the main content scoring; a blinded human expert rescored only a sample, with intraclass correlation of 0.81.
- The study measured emphasis between medication and selected self-care categories, not overall clinical accuracy or appropriateness.
- The author disclosed relevant advisory, equity and editorial roles; NIMH grants supported the work and the funder reportedly had no study role.
- Replication is needed across models, languages, prompts, clinical conditions and independent evaluation teams.
What to watch next
- Independent preregistered replication across open and closed models
- Expert-scored multilingual and crisis-sensitive scenario banks
- Audits that compare signed deployment configurations with the evaluated system
- Human studies measuring comprehension and choice without compromising care
- Disclosure rules for material inference-time controls in patient-facing systems
Living evidence record
Impact record IAI-1L8V21Z
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
3 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what JAMA Network Open published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 3 November 2026
Research question, topics and regions
Whether inference-time activation steering can invisibly shift the balance of medication and self-care content in an open model's answers to synthetic depression-treatment questions
Topics: Mental health · Activation steering · Health information · Model auditing · Depression
Regions: United States · North America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.