How To Judge Tesla FSD: The Validation Stack That Matters

The weakest way to judge Tesla FSD is by watching one impressive drive. The second weakest way is by watching one embarrassing failure. Both can be real and both can mislead. Autonomy has to be judged as a statistical system operating across millions of miles, not as a highlight reel. The better question is whether Tesla's system is improving across a validation stack: perception, prediction, planning, control, intervention frequency, regression management, and operational limits. The stack matters because a vehicle can appear smooth in ordinary traffic while still failing rare cases that dominate safety risk. Thesis: FSD progress becomes credible when improvements are broad, measurable, and durable across software versions, not when one version produces a better demo route. The Five-Layer Validation Stack Layer Question Strong signal Perception Does the car understand the scene? Robust object, lane, signal, and free-space understanding in bad lighting and unusual geometry. Prediction Does it anticipate human behavior? Correct handling of hesitation, aggression, occlusion, pedestrians, cyclists, and construction workers. Planning Does it choose the right path? Decisive but comfortable maneuvers with legal and socially acceptable timing. Control Does it execute smoothly? No harsh braking, lane wobble, curb risk, or awkward stop placement. Regression Does an update keep old skills? New versions improve edge cases without breaking common routes. Why Anecdotes Are So Tempting Autonomy is visible. A person can sit in the car, record a video, and immediately feel that the future has arrived. That is powerful. It is also incomplete. A route that feels magical may not include rare events. A failure clip may show a real bug but not tell us how common it is. Good validation has to combine qualitative evidence with large-sample measurement. The qualitative side explains what the system is learning. The quantitative side tells whether the learning is reliable. Bad metric alone One video Great for product feel, weak for safety conclusions. Better metric Intervention rate Useful when normalized by route type and conditions. Best direction Regression dashboard Shows whether new versions keep old capabilities. The Edge Case Problem Most driving is routine. Autonomy fails in the tail: a truck blocking a lane, a police officer waving traffic through a red light, a child near a curb, temporary construction markings, sun glare, or a cyclist behaving unpredictably. Tesla's fleet gives it a strong data advantage in finding these cases, but finding them is not the same as solving them. A rare case becomes valuable when it enters a training loop, gets labeled or learned from, improves the model, and then passes regression tests without damaging common behavior. This is why the data engine matters as much as the car's behavior on a single route. What a serious FSD scorecard would include Miles per critical intervention, route difficulty, weather and lighting, vulnerable road-user encounters, construction-zone performance, unprotected turn success, emergency vehicle behavior, regression rate by release, and the percentage of fleet miles covered by the latest model. Bottom Line Tesla FSD should be judged by repeatability under variation. The vision-only strategy is compelling if it keeps improving across geography, weather, traffic culture, and software versions. The proof is not one great ride. It is a declining rate of serious interventions across increasingly messy roads.