One of the subtler but I think most noteworthy announcements from this week’s Tesla conference call concerned the ongoing development of Tesla’s Autopilot/”Full Self-Driving”* firmware suite. Before we get into that, though, let’s go back a bit to include some historical context.
Previously, two years ago, Tesla “labelers” were labeling pictures taken by Tesla vehicle cameras in order to train the vehicles to respond to different things (animals, street oddities, other cars, etc.) in certain ways. Well, they were training the Autopilot/Full Self Driving firmware, and then Tesla would occasionally implement over-the-air firmware updates for consumer vehicles (like you get with a smartphone or computer). At some point, seemingly around the middle of 2019, the Tesla Autopilot team determined that this approach was seeing diminishing returns and couldn’t get Tesla to truly self-driving vehicle capability. That is when Tesla made a big shift and spent months doing a ground-up rewrite of the Autopilot code.
(*Side note: Autopilot is the broader term covering all of the passive and active driver-assist features in a Tesla. “Full Self Driving” is a specific suite of features that consumers have to pay $10,000 more for — or $6,000 more for when I bought my Model 3. Also, when being implemented in the car by the driver, with all of these extra features, you say you are “turning on Autopilot.” Many Tesla owners — like me — have had “Navigate on Autopilot” for highways available in our cars for a long time, and a few thousand owners now have a beta version of the most advanced features that basically provides the same Navigate on Autopilot functions for use on city streets, including making turns and passing other cars as needed.)
The big shift was to start labeling videos across time — “4D” labeling as Elon has called it. So, for example, as I understand it, when a car is driving by a fire hydrant, a Tesla labeler would label that so that other Teslas could identify fire hydrants as they drive by them in the future. That is a simple example, but something more obscure may be a plastic bag blowing in front of the car. Ideally, if this is done well, any time a plastic bag blows in front of a Tesla, the car should identify it as such and not brake to avoid hitting it.
The news from the Q1 conference call this week is that Autopilot/FSD labelers are increasingly working … on Autopilot. As in, they are increasingly just checking the auto labeling that the neural nets are doing, rather than doing the labeling themselves. They train the system to label 8 video feeds and then check the auto labeling. And the system seems to be getting better and better at it.
To be honest, some years ago, I would say that many people thought Tesla was doing what it is starting to do now. I know several years ago, when I learned about Tesla using neural nets and having the ability (in theory) to use “shadow Autopilot” to collect an enormous amount of data from most of the Tesla vehicles on the roads, I thought Tesla was already doing this. Many others did too, based on comments I saw on forums and in comments back then. Apparently, though, our expectations for what was happening were some years ahead of reality. Nonetheless, it’s big news that Tesla is sliding into this method more and more now, and this increasingly means that AI will improve Autopilot and hopefully help get the firmware to the 99.999999% safety level Elon is aiming for.
When I got to converse with Elon about the Tesla Autopilot team a bit last year (see: “Tesla Autopilot Innovation Comes From Team Of ~300 Jedi Engineers — Interview With Elon Musk“), he noted that there were “just under 200 [engineers] on the software side and a little over 100 on the chip design side,” but that there were several hundred more people dedicated to labeling.
“We also have over 500 highly skilled labelers,” Elon said. “This is a hard job that really does require skill and training, especially with 4D (3D plus time series) labeling.” But that was just at that moment. The plan was to grow that team, double it.
“We are expanding to 1000 highly skilled labelers. Emphasis on highly skilled. Like I said, this sounds easy, but is actually hard and requires talent.“
This is practically the only time I’ve seen or heard him talk about the labelers and quantify them, so it was one of the most interesting pieces of new information about Tesla or autonomous driving development for me last year. I’m curious now how the evolving system of labeling is affecting that team. Is the team going to shrink again due to AI taking on more and more of the work? Or does Tesla still need so many labelers even as the AI does more work?