I started digging into the safety research around self-driving cars expecting to find the usual suspects — sensor failures, hacking risks, maybe a few bad decisions at crossroads. What I found surprised me. The dangers aren’t the ones you see in the headlines, they’re a cardboard box on the side of the road making a car lock up.
A whistleblower who quit after four months. And an AI that’s too careful in exactly the wrong way.
In 2025, the National Highway Traffic Safety Administration stated: No vehicle currently for sale is fully automated or ‘self-driving.’ Every car you can buy in the US right now still demands your full attention at all times. Yes, modern driver assistance features — automatic emergency braking, adaptive cruise control, lane keeping, save lives. But none of them make the car the driver.
Around 40,000 people die in motor vehicle crashes every year in this country, and the stakes for getting this right are enormous. In 2025, NHTSA released domestic exemptions for the automated vehicle exemption program, marking a key regulatory action. NHTSA also enforces a Standing General Order requiring reporting of crashes involving automated driving systems.
Key Takeaways
UC Irvine researchers found that a cardboard box, bicycle, or traffic cone on the roadside can trigger an autonomous vehicle to permanently stop in the middle of an empty intersection or refuse to change lanes for a nonexistent threat — a “physical denial-of-service” caused by an overly cautious planning module.
A former Waymo test driver quit after four months, citing a near head-on collision with a Toyota Corolla, sudden acceleration to 37 mph in a parking lot, and an abrupt stop that forced an Old Dominion tractor-trailer to barely avoid a crash — and was told by supervisors “well, nobody’s gotten killed by them yet.”
Current autonomous systems aren’t trained well enough to handle rare, high-risk edge cases, and collecting more real-world data won’t solve it because deliberately seeking dangerous driving conditions is costly, slow, and risky.
Table of Contents
The AI that’s too safe for its own good
The conventional worry about self-driving cars is that they’ll be reckless — barreling through intersections, ignoring traffic rules. The problem is the exact opposite: the planning module is so cautious it can create real hazards.
Researchers at University of California, Irvine built a testing tool called PlanFuzz that automatically finds vulnerabilities in widely used automated driving systems. They tested three different behavioral planning implementations from Apollo and Autoware, two of the most prominent open-source autonomous driving platforms.
Put a cardboard box or a bicycle on the side of the road, and the test vehicles would permanently stop — dead halt, on empty thoroughfares and intersections. Not a “slow down and proceed carefully” situation. A full stop, indefinitely. In other cases, the cars refused to change lanes because they perceived a threat that simply wasn’t there.
Lead author Ziwen Wan, a University of California, Irvine Ph. D. student in computer science, put it plainly: the planning module is designed with an abundance of caution but can err on the side of being overly conservative. But that caution can backfire.
Qi Alfred Chen, the University of California, Irvine professor who co-authored the paper, and the team showed that both scenarios cause the same erratic behavior — it’s a physical denial-of-service attack via roadside objects. Someone could intentionally place ordinary objects to mess with the car, and the vehicle can’t tell if that box was left by accident or on purpose.
GeekExtreme’s own deep-dive into this paper is worth reading if you want the full technical breakdown, but the takeaway is this: the primary risk isn’t an AI that’s too aggressive. It’s an AI that’s too conservative in exactly the wrong way.
The software bugs that test drivers fear
A former Waymo test driver spoke with WSMV4 Investigates about his experience, and the incidents he described include a near head-on collision, sudden acceleration to 37 mph, and an abrupt stop that forced a tractor-trailer to barely stop. He worked as a test driver for only four months late last year before quitting. The Metro Nashville Police have been involved in reviewing some of these incidents, raising a critical legal question: who is at fault if an autonomous vehicle crashes?
- A near head-on collision with a white Toyota Corolla.
- Sudden acceleration to 37 mph in an apartment complex parking lot, nearly hitting a dumpster.
- An abrupt stop in moving traffic that forced an Old Dominion tractor-trailer to barely stop in time.
- A Waymo traveling the wrong way down a street in Tempe, Arizona.
- In Santa Monica, a Waymo driverless vehicle struck a child near an elementary school in January.
The driver told investigators that the car “would start doing more sporadic, dangerous stunts,” citing near head-on collisions, sudden accelerations, and a tractor-trailer incident that caused him to quit. When asked if he raised safety concerns with supervisors, his account of the response is the gut punch: He raised safety concerns with supervisors who responded ‘well, nobody’s gotten killed by them yet.’ He quit, telling the news outlet that “the people of Nashville need to know these vehicles are dangerous before someone gets hurt or killed by one.”
Waymo has generally denied these claims, but the specificity of the whistleblower’s account — locations, vehicle types, timestamps, makes it hard to dismiss as an outlier. The supervisor quote alone should give anyone pause. Cybersecurity is a critical issue for AVs, and both USDOT and automotive companies are working to address it for the future safe deployment of these technologies.
The human driver problem no one talks about
The most dangerous part of a self-driving car might be the human driver behind it—an ironic reality laid bare in an unflinching look at the 10 disadvantages of self-driving cars.
Jing Feng, an associate professor in the Department of Psychology at NC State, has researched what happens as automation increases: drivers disengage. Hands come off the wheel. Eyes drift from the road. Minds wander. And that creates a secondary danger that’s rarely factored into safety calculations — a blind spot that highlights the importance of autonomous vehicles.
Remember the Old Dominion tractor-trailer incident from the whistleblower account? The Waymo stopped abruptly in moving traffic, and a semi truck had to brake hard to avoid rear-ending it. When an autonomous vehicle is overly cautious — slamming the brakes for a traffic cone on the shoulder, the human drivers behind it aren’t expecting that behavior, and recent accident data shows that AI errors have caused at least 25 deaths in self-driving cars alone. They’re not calibrated for an AI that stops dead in the middle of a clear road.
The mismatch between AV cautiousness and human driver expectations creates situations where people attempt risky maneuvers — swerving lanes, accelerating around the stopped vehicle, that can cause more harm than the original “threat” ever would. The car’s safety programming doesn’t account for the humans it’s sharing the road with.
The edge cases autonomous cars can’t handle
More data, more training, more refinement. But there’s a fundamental problem: the “long tail” of rare, high-risk scenarios.
A vehicle might navigate a thousand gentle curves flawlessly, then fail on a single sharp bend taken slightly too fast on a wet road. Current autonomous systems aren’t trained well enough to handle these edge cases reliably. collecting more real-world data doesn’t fully solve the problem, because deliberately seeking dangerous driving conditions is costly, slow, and risky. You can’t just send a test fleet out to find black ice and sudden slope changes.
Researchers at Dublin City University and the University of Birmingham are tackling this with a clever approach. They’re using generative AI trained on real driving data to create realistic, demanding high-risk scenarios for simulation training. Think of it as a way to manufacture dangerous situations in a virtual environment, training the system on edge cases it might never encounter in a million miles of normal driving.
The same research team also developed a human-machine shared driving platform that adapts control based on risk. Less experienced drivers on complex or winding roads get timely support from the system, reducing the risk of losing control. It’s a bridge solution — not full autonomy, but smarter assistance that knows when to step in.
The edge case problem remains one of the hardest unsolved challenges in the field, and this approach is still research-stage — not deployed in commercial vehicles.
The trolley problem is already happening
For self-driving cars, the “trolley problem” is not hypothetical — it’s a real programming challenge with documented consequences.
The industry’s initial approach — “just stop”, has proven dangerously naive. When you stop abruptly for a perceived hazard, you create the secondary dangers we talked about earlier.
Veljko Dubljevic, an associate professor of philosophy and science, technology and society at NC State, frames this as the alignment problem — how to ensure AI systems benefit society while realizing human values. His NeuroComputational Ethics Research Group has proposed the Agent-Deed-Consequence (ADC) model as a framework for how AVs should handle these unavoidable crash scenarios. The idea is that sometimes, following the rules strictly isn’t the most ethical choice, and a self-driving car needs to understand that nuance.
Every time an AV brakes hard for a nonexistent obstacle, it’s making an ethical judgment that prioritizes one kind of safety over another. Recent incidents involving pedestrian injuries highlight the real-world stakes of these decisions. The problem is that nobody agreed on how those tradeoffs should be made, and the cars are already on the road making them.
Bonus: the hidden bias in the algorithm
Studies show computer vision systems in AVs may be less able to detect people of color and children. The problem isn’t the sensors — it’s the training data. An equity analysis of these systems reveals that the same detection disparities that put pedestrians of color at higher risk could also widen transportation inequities, as autonomous ride-hailing services may avoid or under-serve neighborhoods with less diverse training data. In one study, pedestrian detection accuracy for darker-skinned individuals was nearly 5% lower than for lighter-skinned individuals, a gap that compounds over millions of miles driven.
Vision model performance depends on the demographics of the data used to train it. If the training set
People Also Ask
What are the dangers of driverless cars?
The biggest dangers aren’t reckless driving but the opposite: overly cautious AI that can freeze in place for a cardboard box or traffic cone, creating real hazards for human drivers behind it. Software bugs, edge cases the system wasn’t trained on, and human drivers who disengage or react unpredictably to strange AV behavior all compound the risk.
Can a cop pull over a Waymo car?
Yes, but it’s complicated. Autonomous vehicles can be programmed to respond to emergency vehicles and pull over, but the real legal question is who’s at fault when something goes wrong — the manufacturer, the software, or the human occupant. Police in cities like Nashville have already been involved in reviewing incidents involving Waymo vehicles.
What’s the difference between driver assistance and full self-driving?
Driver assistance features like automatic emergency braking and adaptive cruise control save lives but still require your full attention at all times — no car you can buy in the US right now is fully automated or self-driving. Full autonomy means the vehicle handles all driving tasks without human intervention, and that technology isn’t commercially available yet despite what some marketing suggests.
Are self-driving cars too cautious?
Yes, and that’s one of the most counterintuitive dangers. The conventional worry is that AVs will be reckless, but the real problem is an AI that’s too conservative — slamming the brakes for a nonexistent threat or stopping dead in moving traffic. That overly cautious behavior creates secondary hazards because human drivers behind the AV don’t expect it and may swerve or rear-end the vehicle.
