Autonomous Vehicles

Published on July 11th, 2016 | by Zachary Shahan

23

What Does Tesla Autopilot “Beta” Mean?

July 11th, 2016 by  

autopilot_handsoff

My hands while testing Autopilot in a Tesla Model X. Photo Credit: Kyle Field.

Mike Barnard, an expert in robotics, might follow this up with a deeper look into the topic of “beta” as it regards Tesla Autopilot and other matters —  to supplement his articles on why Tesla Autopilot is better than driver-assist features in competing brands’ cars and why Tesla’s approach to self-driving cars is better than Google’s and dramatically outdistances Google’s autopilot miles — but since Tesla Motors CEO Elon Musk went on a tweetstorm about this yesterday, providing several interesting comments, I wanted to run down them in a quick summary for clarification.

I think the best approach in this case will be a simple bullet-point list, but first a little background: As Wikipedia nicely summarizes, “beta” in regards to technology is “the last testing release (or the preview release) in the software release life cycle before the ‘release’ version…. Typically it is the last version before a version of a software is fully released to all its actual customers.” That seems clear, right? But when does a product actually cross the line from “beta” version to “release version?” Furthermore, does “beta” mean a technology isn’t ready for the masses, or just that the company wants to reach certain milestones (in use, experience, data, cost, etc.) before removing the label?

I think Elon’s tweets make it very clear what the story is with Tesla’s use of the term, so let’s just jump in. It all started with this article and tweet:

Well, actually, it started with commenters obsessing over the term “beta” in comments and forums around the web (and perhaps the real world), which I’m sure is what led to this article. And even before that, it obviously started with Tesla’s use of the term “beta” for Autopilot. But let’s remember: Tesla chose to use the term, nobody forced it to, and owners don’t just agree to use “beta” Autopilot — they pay extra to do so!

As one CleanTechnica commenter noted recently, if anything, Tesla is being more responsible than other automakers by using the term “beta,” as it is communicating clearly to customers that this is not a final product and is undergoing rapid refinement (and that’s on top of all of the other disclaimers and warnings Tesla is constantly pushing — such as the fact that drivers need to remain alert and ready to take over the Autopilot-controlled vehicle at any time).

On that point, following up on the statement that Autopilot is in “beta” until it reaches 1 billion miles, Elon added: “Use of word ‘beta’ is explicitly so that drivers don’t get comfortable. It is not beta software in the standard sense.”

Ironic that “haters” and people looking to profit on the controversy are twisting the point to make it seem like Tesla is the one being irresponsible. (Let’s remember, that although Tesla’s competitors don’t call their driver-assist features as being in beta, they are essentially trying to do the same thing — they’re just much worse at it.)

In response to a question, Elon added that Autopilot will probably hit 1 billion miles of real-world use in ~6 months, “Will include hundreds of refinements to handle rare corner cases in Autopilot.”

Interestingly, Elon did receive criticism from a top tech tweeter (who I can’t say I’ve ever heard of before), who simply responded to the original tweet at the top with “this is a bad tweet.” Elon, who seemingly values this person’s judgement and didn’t see any reason to support such a claim, responded with one word, “why?”

This is where we start getting into meta discussions about beta, which I’ll leave for Mike to pursue if he feels inclined, but here’s where this person (SecuriTay) goes with it (two tweets combined into one here):

  • “it sounds like you’re retroactively changing the meaning of a term with an arbitrary cutoff point that removes human judgement. Humans should be highlighted as judging fitness for safety-critical designs, not an odometer reading. It looks bad to public.”

Elon’s response makes it sound like that was in line with his approach anyway, and he had just oversimplified for Twitter (I can’t imagine he’d simply say, “okay, we hit 1 billion miles, so beta is over,” so I see no reason to think he didn’t already consider the point SecuriTay made and wasn’t treating “beta” as such a black and white topic related to a nicely rounded number of 1 billion miles — why not 1 billion and 17, after all?). His succinct response:

  • “With less than 1B miles, there simply isn’t enough data. 1B is a necessary but not necessarily sufficient condition.”

Two more tweets later on, to SecuriTay and another tweeter, further clarified:

The internet (have you visited?) is full of armchair experts who would be better off eating a sandwich than making strong claims on social media, but alas, we are not out-of-beta autonomous thinkers. An Austrian journalist took to Twitter with this claim:

  • “What an astounding, monstrous stupidity. Untested components are meant for the lab, not the street.”

Oy… of course Tesla tested it. Tesla heavily tested it. And Tesla even had the hardware installed on consumer cars collecting data for millions of customer-driven, real-world miles before consumers could buy and use Autopilot. But yes, given that it is improving via deep learning and we live on Earth, not in a computer simulation where you can be “sure” it will work perfectly in every single case, it has to be installed and used by customers to get better. Elon’s response:

  • “It is extensively tested in the lab and in the Tesla test fleet. However, there is no substitute for real world experience.”

At the risk of beating a dead version of the automobile’s predecessor here (which I prefer not to do), I feel like I need to reiterate one more time that other automakers also offer driver-assist features, but do not indicate they’re in beta even though they are far inferior to Tesla’s driver-assist features. But I guess it could be said that they don’t need to since the limitations are so obvious to drivers.

As Tesla noted in its blog post about the initial driver death while using Autopilot, and I summarized here: “It is very sad to find out from Tesla Motors that the first Tesla Autopilot fatality has been logged. It comes after 130 million miles, and Tesla noted in its press release that a fatality occurs, on average, every 94 million miles in the United States, and every 60 million miles worldwide.”

Similarly, yesterday, Elon tweeted: “US is ~11 deaths per billion miles. WW is 17 deaths per billion. Autopilot already much better than either so far.”

But a key point expressed elsewhere in Elon’s tweets, in my initial post on the topic, and in Mike Barnard’s “get the facts straight” post, we can’t confirm with statistics at this point that Autopilot driving is safer than non-Autopilot driving. It seems obvious that it is, but sometimes our minds don’t read stories clearly, and we need Tesla to at least reach 1 billion miles of Autopilot driving (in ~6 months) before we can start legitimately discussing whether statistics clarify that it is safer … and whether it should still hold the “beta” tag.


Check out our new 93-page EV report, based on over 2,000 surveys collected from EV drivers in 49 of 50 US states, 26 European countries, and 9 Canadian provinces.

Tags: , ,


About the Author

is tryin' to help society help itself (and other species) with the power of the typed word. He spends most of his time here on CleanTechnica as its director and chief editor, but he's also the president of Important Media and the director/founder of EV Obsession, Solar Love, and Bikocity. Zach is recognized globally as a solar energy, electric car, and energy storage expert. Zach has long-term investments in TSLA, FSLR, SPWR, SEDG, & ABB — after years of covering solar and EVs, he simply has a lot of faith in these particular companies and feels like they are good cleantech companies to invest in.



  • I would not put too much stock in whether the traditional car companies call something “beta” or not. The use of the term “beta” is mainly a difference between different types of industries and cultural practices. Traditional car companies are not software companies and any car they sell is the final product, so they don’t use terms like “beta”, but autopilot is mainly a matter of refining the software and self-learning and it is being updated to new versions with over the air updates. The sensors are fixed, so the hardware isn’t getting any better, but the software is evolving, so calling this “beta” makes sense.

    As for the debate about the safety of autopilot, we don’t have enough data to make any statistically meaningful statement. When we have 10 billion miles of autopilot use, we might be able to determine whether autopilot is safer than manual driving, but saying that this is the first death in 130 million miles does not prove that it is safer than manual driving. Also remember that autopilot is only used at very low speeds (like parking) and at high speeds on limited access roads. Both of these uses have fewer accidents that general driving, so there is some selection bias in the data. Autopilot is turned off when there is the most statistical chance of having an accident.

    One of the commenters says that he/she doesn’t like to be a “guinea pig,” but there is no way that autopilot and eventually autonomous driving will ever be developed without lots of people being guinea pigs. If you don’t think it is ethical to use experimental technology on the road, then you are basically saying that no new automotive technology should ever be developed.

  • GCO

    I disagree with most of the points made in the article. In particular:

    1) Autopilot safety record.
    1 fatality in 130M miles with it engaged, vs how many in the same vehicles and inherently safer roads (divided highways) without it?
    On one hand, Tesla says its vehicles are much safer than average.
    On the other, it claims AP on highways and favorable conditions is (barely) safer than the fleet of all vehicles, which average ~11 years old in the US, on all roads and conditions (1 fatality every ~100M miles). Doesn’t this show that AP, far from increasing the safety of the Model S, degrades it to basically that of an average decade-old car?

    2) Choice.
    First, Tesla only make it clear its software is “beta” in the disclaimer popping up when drivers want to enable a feature they paid for (how many people will fully read that legalese is another question). Nothing in the wording shown on Tesla’s “Design Studio”, when customers buy the car and/or the Autopilot option, even suggests that it is only experimental.
    Second, anyone on or near an open road becomes a potential guinea pig as well, including other Model S drivers who explicitly declined to participate in that “beta testing”. This is not my understanding of giving people a choice.

    • nitpicker357

      1. If Autopilot had a long-term fatality rate of 1 per 1 billion miles traveled, then having 1 fatality in the first 130 million miles would be unremarkable. If Autopilot had a long-term fatality rate of 1 per 44 million miles traveled, then having 1 fatality in the first 130 million miles would be unremarkable.
      2. I do not get to choose whether other cars on the road have collision avoidance systems. I have been rear-ended twice in the last three years. Where is my choice?

      • Matt

        I own/ride a motorcycle and I don’t think you should have right to not wear a helmet when driving one. If you crash the cost of trying to save you (hospital and long term care) will be covered by tax payers if you do not have the money.

  • Steve

    Zachary, it gave me the chills! Haha
    http://www.ntsb.gov/news/press-releases/Pages/PR20150608b.aspx
    ************************************************************************************************
    “Because there will always be better technologies over the horizon, we must be careful to avoid letting perfection become the enemy of the good.” -NTSB
    *************************************************************************************************
    NTSB Press Release
    National Transportation Safety Board Office of Public Affairs

    NTSB Calls for Immediate Action on Collision Avoidance Systems for Vehicles; Cites Slow Progress as Major Safety Issue
    6/8/2015
    ​WASHINGTON – In a report released today, the National Transportation Safety Board outlined the life-saving benefits of currently available collision avoidance systems, and recommended that the technology become standard on all new passenger and commercial vehicles.

    “You don’t pay extra for your seatbelt,” said Chairman Christopher A. Hart. “And you shouldn’t have to pay extra for technology that can help prevent a collision altogether.”

    NTSB’s Special Investigation Report, The Use of Forward Collision Avoidance Systems to Prevent and Mitigate Rear-End Crashes, stresses that collision avoidance systems can prevent or lessen the severity of rear-end crashes, thus saving lives and reducing injuries.

    According to statistics from the National Highway Traffic Safety Administration (NHTSA), rear-end crashes kill about 1,700 people every year and injure half a million more. More than 80% of these deaths and injuries might have been mitigated had the vehicles been equipped with a collision avoidance system.

    The NTSB has made 12 recommendations over 20 years in favor of forward collision avoidance technologies, including 10 recommendations resulting from an earlier Special Investigation Report in 2001.

    The progress on these recommendations, however, has been very limited. The report notes that a lack of incentives and limited public awareness has stunted the wide adoption of collision avoidance technology.

    Only 4 out of 684 passenger vehicle models in 2014 included a complete forward collision avoidance system as a standard feature. When these systems are offered as options, they are often bundled with other non-safety features, making the overall package more expensive.
    ****************************************************************************************************
    “The promise of a next generation of safety improvements has been used too often to justify inaction,” Hart said. “Because there will always be better technologies over the horizon, we must be careful to avoid letting perfection become the enemy of the good.”
    ****************************************************************************************************

    In the report, the NTSB recommends that manufacturers make collision avoidance systems standard equipment in newly manufactured vehicles, beginning with collision warning systems, and adding autonomous emergency braking once NHTSA completes standards for such braking systems.

    Furthermore, the NTSB recommends that NHTSA develop tests and standards in order to rate the performance of each vehicle’s collision avoidance systems and to incorporate those results into an expanded NCAP 5-star safety rating scale.

    The NTSB is also issuing a companion Safety Alert for consumers and commercial fleet owners that urges them to consider vehicles with collision warning and autonomous emergency braking functions.

    • Bob_Wallace

      Collision avoidance systems will add some cost to the vehicle but that should be quickly repaid by insurance premium savings.

      • Martin Lacey

        In a just and fair society that would be true…

        Sadly I can’t say we live in such a world!

        • Bob_Wallace

          Air bags add cost.

          Insurance companies charge lower premiums for cars with air bags.

          • Martin Lacey

            I very much doubt that assertion. It may have featured twenty years ago when they were new tech, but not now. When was the last time your insurance company lowered your premium year on year?

          • Bob_Wallace

            On this page you can find a list of safety equipment which can gain you a discount on your auto insurance.

            http://www.dmv.org/insurance/safety-equipment-auto-insurance-discounts.php

      • Matt

        Note that while safely systems are cost to point of sale, they save cost for the large society. Think of who pays for hospital when you don’t.

      • Steve

        NTSB requested cars have this technology. This meeting will not be adversarial. It will give the NTSB a deep dive into Tesla’s technology. It will be the meeting of the minds as a collaboration to push this technology further. After this preliminary investigation, Tesla will be considered the standard for auto safety and driver assist technology. No other auto company will have the relationship with NTSB that Tesla will have. Any other company that implements driver assist, will be compared against the Tesla standard.

        • Bob_Wallace

          Other car companies offer collision avoidance systems. Here’s a page that lists some –

          http://www.autotrader.com/best-cars/6-affordable-vehicles-with-collision-warning-systems-188920

          • Steve

            Thanks, Bob. Interesting, I wonder what category these cars fall into. Collision avoidance and driver assist are two different categories? Combined, and you have semi-autonomous? Thanks again for the reference link!

          • Bob_Wallace

            Collision avoidance.

            I think one or more other companies have lane keeping but it’s only usable at low speeds. Stop and go traffic. Tesla’s lane keeping system can be used at highway speeds and has lane changing ability, triggered by the driver flicking the turn signal lever.

            Tesla has collision avoidance and lane keeping. It’s the collision avoidance system that gets the blame in this case.

            The combo of collision avoidance and lane keeping is what Tesla calls “autopilot”.

          • Steve

            Bob, thanks for your reply. Maybe all manufacturers should just call it driver assist. A seat-belt is a seat-belt, driver assist is driver assist.

      • Steve

        Tesla should start their own auto insurance company. With the information they collect, they can do some pretty good risk analysis and assign you a rating every year.

    • GCO

      Collision avoidance/mitigation systems have little in common with autonomous driving. It’s perfectly fine for them to work, say, “only” 99% of the time.

    • Matt

      Just as government had to force makers to include seat beats, likely this is something that they will have to require. When seat belts first came out you had to buy and install them yourself. A first step would be so shift the rating system (as recommended by NTSB) that way only a vehiclewith good collision avoidance systems can get best rating. That way insurance company’s have a way to judge if they should give a discount.

      • Steve

        I think Tesla should start their own insurance company. They already have the data on your driving habits.

  • Harry Johnson

    Gawd I hate Twitter…

  • Pilar8542

    I am making around $6000-$8000 a month with an online job. If you are willing to work easy freelance jobs for 2-5 hrs each day from your couch at home and earn solid payment in the same time… Then this gig is for you… SELF97.COM

  • Pilar8542

    SELF97.COM

Back to Top ↑