What Does Tesla Autopilot “Beta” Mean?

My hands while testing Autopilot in a Tesla Model X. Photo Credit: Kyle Field.

Mike Barnard, an expert in robotics, might follow this up with a deeper look into the topic of “beta” as it regards Tesla Autopilot and other matters —  to supplement his articles on why Tesla Autopilot is better than driver-assist features in competing brands’ cars and why Tesla’s approach to self-driving cars is better than Google’s and dramatically outdistances Google’s autopilot miles — but since Tesla Motors CEO Elon Musk went on a tweetstorm about this yesterday, providing several interesting comments, I wanted to run down them in a quick summary for clarification.

I think the best approach in this case will be a simple bullet-point list, but first a little background: As Wikipedia nicely summarizes, “beta” in regards to technology is “the last testing release (or the preview release) in the software release life cycle before the ‘release’ version…. Typically it is the last version before a version of a software is fully released to all its actual customers.” That seems clear, right? But when does a product actually cross the line from “beta” version to “release version?” Furthermore, does “beta” mean a technology isn’t ready for the masses, or just that the company wants to reach certain milestones (in use, experience, data, cost, etc.) before removing the label?

I think Elon’s tweets make it very clear what the story is with Tesla’s use of the term, so let’s just jump in. It all started with this article and tweet:

Misunderstanding of what "beta" means to Tesla for Autopilot: any system w less than 1B miles of real world driving https://t.co/WG5vnFvegI

— Elon Musk (@elonmusk) July 10, 2016

Well, actually, it started with commenters obsessing over the term “beta” in comments and forums around the web (and perhaps the real world), which I’m sure is what led to this article. And even before that, it obviously started with Tesla’s use of the term “beta” for Autopilot. But let’s remember: Tesla chose to use the term, nobody forced it to, and owners don’t just agree to use “beta” Autopilot — they pay extra to do so!

As one CleanTechnica commenter noted recently, if anything, Tesla is being more responsible than other automakers by using the term “beta,” as it is communicating clearly to customers that this is not a final product and is undergoing rapid refinement (and that’s on top of all of the other disclaimers and warnings Tesla is constantly pushing — such as the fact that drivers need to remain alert and ready to take over the Autopilot-controlled vehicle at any time).

On that point, following up on the statement that Autopilot is in “beta” until it reaches 1 billion miles, Elon added: “Use of word ‘beta’ is explicitly so that drivers don’t get comfortable. It is not beta software in the standard sense.”

Ironic that “haters” and people looking to profit on the controversy are twisting the point to make it seem like Tesla is the one being irresponsible. (Let’s remember, that although Tesla’s competitors don’t call their driver-assist features as being in beta, they are essentially trying to do the same thing — they’re just much worse at it.)

Use of word "beta" is explicitly so that drivers don't get comfortable. It is not beta software in the standard sense.

— Elon Musk (@elonmusk) July 10, 2016

In response to a question, Elon added that Autopilot will probably hit 1 billion miles of real-world use in ~6 months, “Will include hundreds of refinements to handle rare corner cases in Autopilot.”

Probably six months. Will include hundreds of refinements to handle rare corner cases in Autopilot.

— Elon Musk (@elonmusk) July 10, 2016

Interestingly, Elon did receive criticism from a top tech tweeter (who I can’t say I’ve ever heard of before), who simply responded to the original tweet at the top with “this is a bad tweet.” Elon, who seemingly values this person’s judgement and didn’t see any reason to support such a claim, responded with one word, “why?”

This is where we start getting into meta discussions about beta, which I’ll leave for Mike to pursue if he feels inclined, but here’s where this person (SecuriTay) goes with it (two tweets combined into one here):

• “it sounds like you’re retroactively changing the meaning of a term with an arbitrary cutoff point that removes human judgement. Humans should be highlighted as judging fitness for safety-critical designs, not an odometer reading. It looks bad to public.”

Elon’s response makes it sound like that was in line with his approach anyway, and he had just oversimplified for Twitter (I can’t imagine he’d simply say, “okay, we hit 1 billion miles, so beta is over,” so I see no reason to think he didn’t already consider the point SecuriTay made and wasn’t treating “beta” as such a black and white topic related to a nicely rounded number of 1 billion miles — why not 1 billion and 17, after all?). His succinct response:

• “With less than 1B miles, there simply isn’t enough data. 1B is a necessary but not necessarily sufficient condition.”

Two more tweets later on, to SecuriTay and another tweeter, further clarified:

Unless we get there and find it isn't enough. Just won't be before 1B.

— Elon Musk (@elonmusk) July 10, 2016

Tricky in 140 chars, but: 1B needed for min statistical sample size. More software mods and data may be needed beyond that.

— Elon Musk (@elonmusk) July 10, 2016

The internet (have you visited?) is full of armchair experts who would be better off eating a sandwich than making strong claims on social media, but alas, we are not out-of-beta autonomous thinkers. An Austrian journalist took to Twitter with this claim:

• “What an astounding, monstrous stupidity. Untested components are meant for the lab, not the street.”

Oy… of course Tesla tested it. Tesla heavily tested it. And Tesla even had the hardware installed on consumer cars collecting data for millions of customer-driven, real-world miles before consumers could buy and use Autopilot. But yes, given that it is improving via deep learning and we live on Earth, not in a computer simulation where you can be “sure” it will work perfectly in every single case, it has to be installed and used by customers to get better. Elon’s response:

• “It is extensively tested in the lab and in the Tesla test fleet. However, there is no substitute for real world experience.”

It is extensively tested in the lab and in the Tesla test fleet. However, there is no substitute for real world experience.

— Elon Musk (@elonmusk) July 10, 2016

At the risk of beating a dead version of the automobile’s predecessor here (which I prefer not to do), I feel like I need to reiterate one more time that other automakers also offer driver-assist features, but do not indicate they’re in beta even though they are far inferior to Tesla’s driver-assist features. But I guess it could be said that they don’t need to since the limitations are so obvious to drivers.

As Tesla noted in its blog post about the initial driver death while using Autopilot, and I summarized here: “It is very sad to find out from Tesla Motors that the first Tesla Autopilot fatality has been logged. It comes after 130 million miles, and Tesla noted in its press release that a fatality occurs, on average, every 94 million miles in the United States, and every 60 million miles worldwide.”

Similarly, yesterday, Elon tweeted: “US is ~11 deaths per billion miles. WW is 17 deaths per billion. Autopilot already much better than either so far.”

US is ~11 deaths per billion miles. WW is 17 deaths per billion. Autopilot already much better than either so far.

— Elon Musk (@elonmusk) July 10, 2016

But a key point expressed elsewhere in Elon’s tweets, in my initial post on the topic, and in Mike Barnard’s “get the facts straight” post, we can’t confirm with statistics at this point that Autopilot driving is safer than non-Autopilot driving. It seems obvious that it is, but sometimes our minds don’t read stories clearly, and we need Tesla to at least reach 1 billion miles of Autopilot driving (in ~6 months) before we can start legitimately discussing whether statistics clarify that it is safer … and whether it should still hold the “beta” tag.