Can a driving AI refuse an unsafe instruction?
In new simulated-driving tests, a model designed to reason about conflicts cut collisions from 60 to 38 versus the strongest safety-prompted baseline. The manuscript does not establish public-road safety or a formal guarantee.
The short version
3 key points
- 1.On 150 conflict routes, SafeDriveVLA recorded 38 collisions, nine traffic violations and four out-of-route events, compared with 60, 61 and 11 for the strongest safety-prompted SimLingo baseline.
- 2.On 210 traffic-free routes where every instruction should be followed, average language-command compliance was 83.0%, versus at most 55.6% for the comparison system, but speed error was substantially worse.
- 3.The evidence comes mainly from the CARLA simulator and fixed instruction pools; it is not a road trial, certification result or formal safety guarantee.
The answer: the model refused more dangerous requests in simulation
In the researchers' closed-loop simulator tests, a driving model that explicitly predicted whether to follow, delay or override a navigation signal handled unsafe instructions better than the comparison system. Across 150 conflict routes, SafeDriveVLA achieved a driving score of 85.9 and a 67.3% success rate. It recorded 38 collisions, nine traffic violations and four out-of-route events. The strongest safety-prompted SimLingo comparison recorded 60 collisions, 61 traffic violations and 11 out-of-route events, with a driving score of 72.8 and a 38.0% success rate.
That is meaningful benchmark progress, not proof that the system is safe enough for a vehicle. The study was built to expose a specific weakness: language-capable driving models may obey a navigation request without grounding it in the scene. A request to keep speed can become dangerous when a vehicle blocks the lane; a turn command can point into oncoming traffic. The proposed system performed better at resolving those conflicts while remaining able to follow safe instructions, but its decisions inherit simulation limits and model error.[1]
Why following every instruction is not the goal
End-to-end vision-language-action models combine camera input, a navigation signal and action generation. Their language interface can accept open-ended instructions rather than only a small fixed command set. That flexibility creates a helpfulness-versus-safety problem familiar from language models: a system that refuses everything looks safe but is useless, while a system that follows every instruction can drive into a hazard. Existing driving benchmarks often measure one side without the other.
The authors therefore built separate tests for compliance and conflict. CARLA-F contains 210 traffic-free routes with green lights, so every safe command should be executed. B2D-C contains 150 routes where a language instruction becomes unsafe at a safety-critical event. NavSim-C contains 300 real-world-sensor frames whose discrete command is replaced with a turn that would cause a collision or leave the drivable area. B2D-Adv adds 15 routes with instructions intended to override the model's safety prior under any circumstance.[1]
How SafeDriveVLA makes the decision visible
The system labels expert-driving frames with one of three modes. Strict means the maneuver can be executed immediately, cautious means it should be followed while yielding to local hazards, and fallback means the navigation signal should be overridden because no safe action remains. During training, those labels are inferred from 2.5 seconds of expert trajectory using kinematic rules. At inference, the model predicts the mode before generating action tokens; it does not receive the future expert path.
A second component is a frozen latent world model. It takes the current scene and rolls it forward for 2.5 seconds under an action anchor representing strict execution of the requested maneuver. Those predicted latent states are fed to an InternVL3-1B vision-language backbone alongside the camera view and navigation signal. The intention is to let the policy preview the consequence of obedience. The world model was trained for 20 epochs on half of SimLingo's collected data, while the driving model was fine-tuned for 14 epochs on eight RTX PRO 6000 Blackwell GPUs.[1]
The model improved both compliance and conflict handling
On traffic-free CARLA-F routes, SafeDriveVLA followed natural-language navigation with an average compliance rate of 83.0%. The best SimLingo variant reached 55.6%. The difference was largest for lane changes: the comparisons completed none in either direction, whereas SafeDriveVLA reached 63.6% for left and 72.4% for right. On the standard Bench2Drive task, its waypoint-based configuration reached an 88.16 driving score and 68.64% success rate, modestly above the matching SimLingo results averaged across waypoint and command interfaces.
The conflict benchmark showed a larger separation. Prioritizing instruction following made SimLingo deteriorate to a 56.3 driving score, 12.7% success and 166 collisions. A safety prompt restored its score but did not solve the problem. SafeDriveVLA reduced collisions by 36.7%, traffic violations by 85.2% and out-of-route events by 63.6% relative to the safety-prompted variant. On the 15 adversarial routes, its driving score fell from 94.7 without the attack to 83.9 with it; SimLingo fell from 95.7 to 76.0.[1]
The ablation identifies a real trade-off
Removing the new components helps show what each contributed. Adding the three-way mode token to the shared backbone improved the conflict driving score from 79.2 to 87.3 and cut traffic violations from 47 to 10, but safe-instruction compliance rose only from 55.1% to 59.8%. Adding world-model predictions lifted compliance to 83.0%, while conflict performance eased slightly to 85.9. The authors interpret the mode token as making the model cautious and the imagined future as helping it act when following is actually safe.
There was a cost: speed control worsened. SafeDriveVLA's mean speed error on CARLA-F was 4.28 metres per second, compared with 1.31 to 1.67 for the SimLingo variants. The action anchors represented maneuvers but not requested speed, so every speed instruction was imagined with the same follow-road anchor. This is not a minor presentation detail. A system that avoids one class of conflict while poorly matching speed has not solved the wider control problem.[1]
What the real-world diagnostic did—and did not—show
NavSim-C used 300 curated frames from real-world sensor data to test existing models after replacing the original command with an unsafe left or right turn. Every evaluated model degraded. AutoVLA's composite score fell from 88.2 to 70.7; the strongest comparison, CuriousVLA, still fell by 6.2 points to 84.4. That supports the authors' claim that command-scene conflict is not confined to one model family.
SafeDriveVLA itself was not evaluated on NavSim-C because it was trained for single-view CARLA imitation, while most models in that comparison use multiple views and reinforcement learning tuned to the benchmark. The real-world frames therefore diagnose a broader failure mode; they do not provide real-world validation of the proposed fix. No vehicle carried the system on public roads, and the study reports benchmark events rather than injuries, near misses, passenger behaviour or interactions with unpredictable road users.[1]
Limits that matter before deployment
The mode labels were generated by kinematic rules, not annotated driver intent, so strict, cautious and fallback can be wrong or oversimplified. Instructions came from fixed pools of paraphrases rather than the full variety of human speech. The world model predicts a plausible latent future, not a physically certified counterfactual, and it does not formally guarantee collision avoidance. It can also fail to anticipate how pedestrians or other drivers react to the ego vehicle.
Most evaluation was in CARLA, where sensing, traffic behaviour and edge cases differ from roads. Counts such as 38 versus 60 collisions are route-event totals in a benchmark, not estimates of real collision risk. Comparisons were limited by which systems accept free-form language and by different training or navigation interfaces. The arXiv manuscript says it was accepted at CoRL 2026, but the final proceedings record was not independently checked. The work was supported by US NSF and Department of Transportation grants.[1]
What would change the assessment
Confidence would rise with independently reproduced results across simulators, weather, maps, sensors and model backbones; preregistered tests using unseen free-form instructions; and evaluation of calibrated uncertainty. Speed-aware action conditioning is an immediate technical requirement. Conflict labels should be reviewed by expert drivers, and results should separate unavoidable benchmark events from errors caused by obeying the instruction. Testing also needs multilingual, ambiguous and conversational requests rather than only paraphrased commands.
Before public-road use, the decision layer would need safety engineering outside the neural model: a constrained operational domain, monitoring, a verified fallback, transparent override rules, incident logging and staged trials beginning on closed tracks. Human-factors studies should ask whether occupants understand refusals and whether adversarial or inattentive speech can distract the system. For now, the study's contribution is a better test and a promising architecture. It does not show that a language-directed vehicle can safely arbitrate every conflict in the real world.[1]
What this means for people
- Drivers and passengers need a vehicle to reject dangerous requests without becoming so cautious that it fails ordinary navigation.
- Safety engineers gain a paired test for obedience and refusal, but still need independent validation and non-neural safeguards.
- Road users are not represented by simulator collision counts; public-road safety, human reactions and rare hazards remain untested.
Global context
The research team spans universities in the United States and Singapore, while the core driving tests use CARLA and Bench2Drive rather than a national road system. A real-world-sensor subset showed that conflict sensitivity affects several model families, but the proposed system itself remains simulator-tested.
What the evidence does not yet show
- Primarily closed-loop simulation, with no public-road or closed-track vehicle trial
- Rule-derived mode labels rather than observed driver intent
- Fixed paraphrase pools do not represent open-ended speech or multilingual use
- World-model predictions can be wrong and provide no formal safety guarantee
- SafeDriveVLA was not evaluated on the real-world-sensor NavSim-C subset
- Substantially higher speed error than the comparison models
What to watch next
- Independent replication with unseen language, maps, weather, sensors and model families
- Speed-aware action anchors and calibrated uncertainty for imagined outcomes
- Closed-track trials with verified fallback systems before any public-road evaluation
- Human-factors testing of refusal explanations, distraction and malicious passenger instructions
Living evidence record
Impact record IAI-10D6NX6
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
15 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 15 October 2026
Research question, topics and regions
Whether an end-to-end driving model can both follow safe natural-language navigation and override instructions that conflict with the road scene
Topics: Autonomous driving · Vision-language-action models · Safety evaluation · Adversarial instructions · Closed-loop testing · World models
Regions: Global · North America · Asia · United States · Singapore
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
AI basics · Background explainer
Published 2018-11-28 · 2 min 31 sec
What is artificial intelligence? | The Royal Society
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
AI Risks & Safety
Are disabled people shaping health-AI research?
A review of 93 studies found only six used co-design, usability testing or user validation with disabled people. The evidence maps a participation gap, not the performance of every disability-focused AI tool.
9 min · 1 source
AI Risks & Safety
Can small AI models replicate across ordinary devices?
Anthropic's controlled evaluation shows that advanced models can assemble inference on varied hardware and that GLM-5.2 completed a simplified eight-node replication chain. It did not demonstrate an autonomous worm escaping onto the internet: access was deliberately easy, the agent model ran elsewhere, and the near-term timeline is a projection.
7 min · 1 source
AI Risks & Safety
Can connected AI agents turn a benign request into harm?
A Meta-led preprint found harmful outcomes in 192 of 1,000 simulated agent scenarios generated by its new red-teaming method. The result exposes cross-service risk, but it comes from one synthetic environment and model-based judges—not live consumer accounts.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.