A note on Tesla's revised safety report crash rates

Tesla safety reports

Tesla Inc. is one of the largest manufacturers of electric vehicles in the world, with 1.3 million vehicles delivered in 2022 [Tesla (2023), p. 3]. Their SAE Level 2 ADS, Autopilot, has been offered as a feature since 2015 and has accumulated over 3 billion miles of on-road driving (Karpathy, 2020).

Tesla has issued quarterly safety reports since the third quarter of 2018. Tesla uses a standard calendar, with Q1 corresponding to 1 January through 31 March. Tesla's safety reports represent a promising data source where crash data are collected for vehicles both with and without Level 2 automation using a consistent, albeit non-standard, severity threshold using a vehicle population with similar ages, body styles, and safety features. While driving environment differs greatly—prior to November 2022, Level 2 automation was used almost exclusively on freeways (Nedelea, 2022)—this can somewhat be controlled for using data from naturalistic driving studies (Goodall, 2024).

Initially, Tesla reported the average million miles between crashes for three operating categories (Tesla, 2022):

• ‘Autopilot engaged’

• ‘without Autopilot but with our active safety features’

• ‘without Autopilot and without our active safety features’.

Beginning in Q2 2021, Tesla began using different categories (Tesla, 2022):

• ‘Autopilot technology (Autosteer and active safety features)’

• ‘not using Autopilot technology (no Autosteer and active safety features)’.

Initially, Tesla used these new categories for Q2 2021 and later, while keeping the old categories for Q1 2021 and prior. These changes coincided with NHTSA's Standing General Order of June 2021, although it is unclear if the timing was intentional or coincidental on Tesla's part.

According to the Internet Archive's Wayback Machine (Ogden et al., 2023), sometime between 5 January (Tesla, 2023) and 8 January 2023 (Tesla, 2023), Tesla revised all safety reports to match the new two-category system, but changed the category descriptions slightly:

• ‘using Autopilot technology’

• ‘not using Autopilot technology’.

In addition to changing the category names, Tesla also revised their crash rates, even for those with identical definitions such as ‘Autopilot engaged’ and ‘using Autopilot technology’. In the footnotes, Tesla clarified that they had ‘discovered reports of certain events where no airbag or other active restraint deployed, single events that were counted more than once, and reports of invalid or duplicated mileage records rates’ (Tesla, 2023).

While the original crash rates are no longer listed on Tesla's website, rates between Q3 2018 and Q1 2021 were recorded in the literature (Goodall, 2024), and all rates were recorded by the Internet Archive's Wayback Machine (Tesla, 2024), a digital archive of web content (Ogden et al., 2023). The remainder of this note investigates how the crash rates changed as a result of the January 2023 revision.

Autopilot crash rates

Many quarters showed substantially different Autopilot crash rates postrevision, from a 10.8% reduction in crashes per mile in Q2 2020 to a 12.7% increase in Q3 2019 compared to crash rates prior to the revision. The average change was a 3.6% reduction in Autopilot crashes. Prerevision and postrevision Autopilot crash rates are shown in Table 1.

Non-Autopilot crash rates

The second category, ‘not using Autopilot technology’ (hereafter referred to as ‘non-AP’) implies that any crash not in the first category must be in the second category. An analysis of the data, however, raises several questions.

Table 2 and Figure 1 show crash rates for all non-Autopilot categories visible on Tesla's website on 28 December 2022 (prior to the revision) and 24 February 2024 (after the revision).

Note that ‘not using Autopilot technology’ tracks closely with the two categories ‘without Autopilot and without our active safety features’ and ‘not using Autopilot technology (no Autosteer and active safety features)’. This suggests that the new revised category is heavily weighted toward crashes and miles where active safety systems were not running.

This is surprising, as studies have shown that drivers generally leave most active safety features running. The Highway Loss Data Institute (HLDI) conducted a study in 2016 at vehicle service departments (Reagan et al., 2018). Vehicles arriving for service were checked to see if various driver assistance systems were activated. The authors found that 93% of the 659 front crash prevention systems were turned on. HLDI's study was limited to vehicles where the on/off setting was maintained through multiple ignitions cycles, and excluded vehicles where features were reverted to a default status upon ignition. These results indicate that a high percentage of drivers use active safety features such as front crash prevention.

Many Tesla vehicles are equipped with four active safety features: lane assist, collision avoidance assist, speed assist, and cabin camera. Collision avoidance assist consists of several sub-features, one of which is automatic emergency braking. According to the most recent owner's manuals of the Model S, X, Y, and 3 (software version 2023.32), automatic emergency braking is activated by default at the beginning of every ignition cycle, and operates continuously when the vehicle travels at speeds between approximately 3 mi/hr (5 km/hr) and 124 mi/hr (200 km/hr) [Tesla (2023), p. 136; Tesla (2023), p. 154;Tesla (2023), p. 141; Tesla (2023), p. 128]. Automatic emergency braking can only be deactivated by shifting the vehicle into park and selecting Controls > Autopilot > Automatic emergency braking from the vehicle's menu. This procedure must be repeated at the beginning of each drive, as automatic emergency braking is active by default.

Tesla vehicles appear to have at least one form of active safety systems, automatic emergency braking, running by default across nearly all speeds and driving conditions. Even though drivers can deactivate this feature at the beginning of every drive by pushing three or four buttons from the menu, studies show that only 7% actually do so (Reagan et al., 2018), even when their cars remember their selection trip-to-trip (Tesla's do not). This suggests that for nearly all non-Autopilot miles, an active safety feature is running.

If active safety features are active on nearly all Tesla vehicle miles, then one would expect the non-AP crash rate to be very close to the ‘without Autopilot but with our active safety features’ crash rate from the prerevision reports, as these would represent the vast majority of all non-AP mileage. Analysis shows that this is in fact not the case, as the revised non-AP crash rate is much closer to the ‘without Autopilot and without our active safety features’ crash rate. The next section discusses potential reasons for this discrepancy.

Universal quantification

In predicate logic, universal quantification asserts that a property must hold true for all members of its domain (Shapiro & Kouri Kissel, 2022). When Tesla stated that crash rates were for vehicles running ‘without Autopilot but with our active safety features’ (Tesla, 2023), they may have been using a universal quantification with regards to active safety features, and thereby including crashes and miles that occurred only when all active safety feature were engaged. If the vast majority of crashes and miles occurred when at least one active safety system was either disabled by the driver or outside of the system's design domain, then the postrevision crash rates would be biased towards ‘without active safety feature’ crash rates, as was observed in the revised data.

Tesla's safety manuals, however, show that is impossible for all active safety features to run simultaneously. For example, Obstacle-aware acceleration, a sub-feature of Collision avoidance assist, is designed only to work at speeds between 0 and 10 mi/hr (16 km/hr) [Tesla (2023), p. 137], while Lane departure avoidance, a sub-feature of Lane assist, is only operational when driving between 40 and 90 mi/hr (64 and 145 km/hr) ‘on roads with clearly visible lane markings’ [Tesla (2023), p. 132]. Neither system can operate at speeds between 10 and 40 mi/hr, and so universal quantification does not explain the observed bias in the revised rates. Table 3 shows the operational speeds of various Tesla active safety features in software version 2023.32.

Other classification schemes

Tesla may be using a different method to classify non-AP crashes as either with or without applicable active safety features. Although there is no way to determine this from the available data, some possible schemes are:

• Exclusion of certain applicable active safety features. Tesla may be using a universal quantification but excluding specific features, such as Obstacle Aware Acceleration. This would permit all other active safety features to be running in certain instances, thereby classifying them as a small sample of ‘with active safety features’ crashes and miles.

• Different minimum feature thresholds. Tesla may instead consider a crash to be ‘with active safety features’ if two, three, four, etc. features are running simultaneously.

• Differences in model-year vehicles. Active safety features were reviewed for a recent software version in all four Tesla models. There may be earlier software versions on Tesla vehicles with fewer active safety features or with different operational speed or other characteristics that may result in a large percentage of vehicles operating without active safety features per Tesla's definition. Given the numerous software versions, possible variants in crash classification schemes, and uncertainty in crash/mileage sample size, a detailed review of vehicle specifications is unlikely to resolve these questions and was considered outside the scope of this note.

There may be other classifications schemes with consistent logic and utilization at play. Additional context would benefit researchers.

Unreported data

It is possible that some data was not included in the revised crash rates, either as part of the exclusions mentioned in the January 2023 update (Tesla, 2023) or by some other means. This theory has some support under the terminology used for the Q2–Q4 2021 reports, where ‘not using Autopilot technology (no Autosteer and active safety features)’ could be read as ‘no Autosteer and no active safety features’. Under this interpretation, the crashes involving active safety features without Autosteer would not have been counted.

Additional evidence can be found by analyzing the relative weights of combined averages of crash rates. Before the revisions, with and without active safety features crash rates are reported separately, while after the revisions they are reported together. Although precise mileage and crash counts are unavailable, the relative weights of the two prerevision crash rates in the combined postrevision crash rate can be calculated using the formula for average of averages:

Substituting $n_2=1-n_1$ yields:

The sample sizes $n_1$ and $n_2$ represent the relative sample sizes of prerevision crash rates in the postrevision crash rate $\mu$ .

In 8 of the 11 quarters, the revised crash rate is higher than both of the prerevision crash rates, resulting in negative weights. This is probably due to reclassification of crashes as acknowledged by Tesla in the revision. In the remaining three quarters, the crashes with active safety features represented 7% (Q2 2020), 14% (Q1 2020), and 1% (Q1 2021) of the revised crash rate. Focusing on the most balanced quarter, Q1 2020, ‘without active safety features’ represented 86% of the revised crash rate while ‘with active safety features’ represented 14%. As crash rate is defined as crashes per vehicle-miles traveled (and assuming insignificant changes to the number of crashes and miles as part of the revision), then non-Autopilot mileage without active safety features was approximately 6.3 times greater than with active safety features for that quarter. It seems highly unlikely that vehicles traveled 6.3 times further without active safety features than with given that active safety features are activated by default at each ignition cycle.

According to the central limit theorem, as the sample size of a population of means increases, the sample mean variance will decrease (Kwak & Kim, 2017). When applied to Tesla crash rates, and assuming vehicles without active safety features traveled at least 6.3 times further than vehicles with active safety features, one would expect the ‘without active safety features’ crash rates to be more consistent over time than those of vehicles with active safety features due to their larger sample sizes.

Because the quarterly crash reports represent different Tesla software versions in a continuously evolving vehicle fleet, these cannot be considered means from the same population and so a direct analysis of sample mean variance cannot be performed. Surrogate techniques, however, suggest that the supposed larger sample size yields greater variability. First, a visual inspection of Figure 1 does not show an observable difference in variation. Second, the sample standard deviation of ‘with active safety features’ is 7.2 (14.7% coefficient of variation), lower than ‘without active safety features’ at 15.1 (21.6% coefficient of variation). This is a counterintuitive result, given that ‘without active safety features’ was expected to have a larger sample size based on the analysis of combined averages, and therefore a smaller coefficient of variation.

The limited evidence suggests that ‘with active safety features’ crashes may have actually been the larger sample before the revisions as indicated by the smaller coefficient of variation and the general prevalence of active safety features on Tesla vehicles. Tesla's crash rate revisions, however, suggest that ‘with active safety features’ crashes were the smaller sample. Simply reclassifying crashes from ‘with active safety’ to ‘without’ would not affect the combined average. Instead, the most efficient way to achieve these results would be to remove or otherwise undercount a portion of the ‘with active safety features’ crashes.

This cursory analysis raises several questions about sample size, crash classifications, and data transparency that require additional context before any conclusions can be drawn regarding the safety and crash rates of Tesla vehicles.