Every six months or so, Xinzhou Wu, Nvidia’s head of automotive, invites CEO Jensen Huang for a ride in a vehicle equipped with the company’s hands-free autonomous driving system. But only if Wu has “good confidence” in the system’s driving capabilities.
Recently, the two went for a drive through downtown San Francisco in a Mercedes CLA sedan from Woodside, California with MB.Drive Assist Pro, a hands-free driver assist system designed in part by Nvidia that is similar to Tesla’s fully self-driving. The mood was light, even if the traffic was quite heavy.
“Tell me when you’re in autonomous mode,” Huang told Wu, according to a video provided of the ride the edge“Then I can worry less about my safety.”
During the 22-minute video, the Mercedes takes Huang and Wu through a series of everyday obstacles, such as construction sites, double-parked cars, and passing lanes lined with rows of orange cones. Nvidia’s system seems quite capable, although the video is edited and not rendered in real time. (Nvidia spokeswoman Jessica Soares later said there were no disconnects during the ride.)
Still, it didn’t seem that different from my own experience last year with Nvidia executives riding shotgun in a Mercedes with the hands-free driving system activated. I was impressed by the system’s ability to handle traffic signals, four-way stops, double-parked cars, unsafe left turns, and all the pedestrians and bicyclists and scooter riders that San Francisco can throw at you. If Tesla can do this with a bit of silicon and a bunch of cameras, it stands to reason that even the world’s most valuable company can figure it out.
‘Chat GPT moment for physical AI’
After years of working behind the scenes, Nvidia is trying to take a more prominent leadership position on autonomous driving. Not only is it supplying chips to companies like Tesla, but it’s also offering its AI-powered driving features to partners like Mercedes, Jaguar Land Rover, and Lucid. At CES earlier this year, Huang unveiled Alpamayo, a portfolio of AI models, simulation blueprints, and datasets that can give vehicles Level 4 autonomy, allowing them to fully drive themselves in certain situations. Huang called the announcement a “ChatGPT moment for physical AI.”
In the car with Wu, Huang is less bombastic and more introspective—but no less enthusiastic about the future of technology. “I think the challenge, of course, is Alpamayo, as incredibly smart as it is – and it can argue about situations – we don’t know what it can’t do,” he said. “And that’s the challenge, and that’s why our classical stack is so incredibly important.”
After years of working behind the scenes, Nvidia is trying to take a more prominent leadership position on autonomous driving.
Huang boasts that Nvidia’s approach to autonomous driving is “unique” because it combines an end-to-end AI model with a traditional, human-engineered “classic” stack. He theorizes that pure end-to-end models are difficult to verify for security. In contrast, the classical stack follows well-established engineering protocols and processes that make it easy to verify that certain behaviors are sufficiently safe. By combining the two approaches, Nvidia’s system can benefit from human-like driving while maintaining a safety framework under traditional rules.
Huang’s claim to a unique perspective in the industry doesn’t quite hold up. Other AV operators also use end-to-end neural networks with explicit safety rules that dictate how the vehicle should respond. But it’s certainly true that end-to-end learning, which is more human-like and less robotic in its driving, is becoming more popular. Waymo relies on a hybrid system, while Tesla relies exclusively on end-to-end neural networks.
In an interview, Wu said the end-to-end models are better able to respond to things like speed bumps or lane changes without feeling mechanical or overly robotic. “That’s why this really is a ChatGPT moment,” he said. “It’s only when your car drives really confidently … then basically customers will feel more willing to use it.”
Tesla and the high cost of self-driving
I asked Wu what he thought of Nvidia’s approach compared to Tesla’s fully self-driving car, which has traveled more than 8.5 billion miles but has been involved in a number of troubling safety incidents, including 23 injuries and at least two deaths. Last December, an Nvidia executive told me the company had tested the two systems against each other. The number of driver takeovers for Nvidia’s systems is comparable, he said, sometimes favoring one system, sometimes the other.
Wu declined to comment directly on Tesla’s safety record, but explained that Nvidia differentiates itself by using multiple sensors, including cameras, radar, ultrasonic sensors, and — in higher configurations — lidar. Nvidia believes that redundancy and diversity in sensing technologies are critical to handling challenging edge cases and achieving high levels of safety, Wu said.
“It’s only when your car drives really confidently … then basically customers will feel more willing to use it.”
— Xin Zhou Wu
Additional sensors mean additional costs. The inclusion of lidar in particular suggests that Nvidia’s safest system will only be accessible to wealthy Mercedes owners. But Wu believes that Nvidia’s vertically integrated approach allows it to deliver the required security performance at the lowest feasible cost.
Nvidia’s DRIVE Hyperion platform is designed with multiple configurations in mind. Relying primarily on cameras and radar, the basic version uses a simpler and more cost-effective sensor setup. These sensors have become dramatically cheaper over the last decade due to mass production. Ultrasonic sensors are already extremely cheap. For higher levels of autonomy, the platform can incorporate lidar sensors, and given the falling cost of lidar, Wu said he believes vehicles costing around $40,000 to $50,000 can actually incorporate the full sensor stack needed for advanced autonomy.
Advantages and disadvantages of data
I asked Wu about recent safety incidents involving Waymo vehicles, such as the company’s robotaxis blocking intersections during a blackout in San Francisco. Nvidia is already running similar edge cases through its simulators, he said. In fact, the company relies heavily on simulated driving data to make up for its losses in real-world testing. Tesla has billions of real-world driving miles, thanks to its vast fleet of customer cars. Waymo has logged nearly 200 million fully autonomous miles on public roads. How can Nvidia ever hope to catch up?
“The big infrastructure game is really simulation,” Wu said. Nvidia is taking two approaches to this. One is Neural Reconstruction, or NuRec, in which the company’s engineers recreate real-world driving scenarios using sensor data collected from vehicles in the field. The second is augmentation, which changes elements within the reconstructed scene to explore different possible outcomes. This allows engineers to test how the autonomous system behaves under slightly different conditions and identify rare edge cases that may occur in the original dataset.
“We can allow a pedestrian to exit faster, slower, in different places,” he said. “This is what we call dataset fuzziness.”
Nvidia receives dashcam footage from its partners to feed the data it uses in simulations. It also recreates edge cases from these Waymo events, such as blackouts, and trains its system to respond without blocking intersections.
But the ultimate goal is to create a system that uses reasoning to avoid these edge-case traps — thus preventing the need for real-world driving data in the first place. Wu’s team is working on something they call the Vision Language Action Model, which will put this theory into practice. These models combine visual perception, language understanding, and physical processes into a unified architecture, building on large foundational models already trained on Internet-scale datasets. Wu likens it to a driver’s aid.
“When we teach a kid to drive, they read the rulebook and then do 20 hours of practice behind the wheel,” Wu said. “Usually, they’re not bad drivers to begin with – although, of course, it takes experience to improve. Ultimately, this is how we want the model to work: in the future, with just a rule book and 20 hours of training data, it will learn to drive.”
