Does training improve users' mental models about adaptive cruise control?

Background

Vehicle Automation technologies have been making rapid advances in terms of development and deployment in the past decades (NHTSA, 2017). The Society of Automotive Engineers (SAE) has classified Vehicle Automation into six levels (SAE, 2021), ranging from no automation (Level 0) to full automation (Level 5). Although fully automated driving (Level 4 and 5) requiring no driver or operator dependency has not yet been achieved, and there are significant technical and human factors challenges to deploying Level 3, there have been positive strides in the development of partially automated driving features (Level 1 and 2), otherwise known as Advanced Driving Assistance Systems (ADAS). These systems carry out various automated functions such as collision avoidance, vehicle speed regulation and gap distance maintenance (Adaptive Cruise Control), or lane position/centering assistance (Lane Centering Assist), etc. ADAS promises safety and convenience benefits to drivers. However, because these systems are designed to take over some of the traditional driving tasks, the introduction of these systems has changed how drivers interact with their vehicles. In vehicles with these systems, drivers assume a new role, i.e., supervising the functions of these systems and monitoring their driving environment for possible intervention situations.

This new role requires drivers to have appropriate knowledge and awareness about their systems' capabilities and, potentially more importantly, their limitations. However, the literature suggests that drivers are unaware or uninformed about their vehicles' systems (Jenness et al., 2008; McDonald et al., 2018). There may be various reasons for this, with a potentially important one related to the resources available for drivers to understand these systems. While studies have shown that many drivers choose to obtain their knowledge about these systems from owner's manuals (McDonald et al., 2016), they only partially or incompletely read these manuals (Mehlenbacher et al., 2002). This is of particular importance since new ADAS users have difficulties understanding their systems (Larsson, 2012). Moreover, the quality of the material presented in the owner's manual can also affect the driver's knowledge and shape their perceptions of these systems (Singer & Jenness, 2020). Also, there are variances and inconsistencies in the reporting of system limitations across different manufacturers (Pradhan et al., 2021), and this could result in misconceptions and overestimation of the systems' capabilities.

There is evidence that a lack of knowledge or any misconceptions about such systems may affect drivers' mental models, which could potentially manifest as action or response-related driver errors (Dickie & Boyle, 2009; McDonald et al., 2018; Pradhan et al., 2021). Mental models have been defined as ‘a representation of the typical causal interconnections involving actions and environmental factors that influence a system's functioning’ (Durso & Gronlund, 1999). Mental models continuously update knowledge stored in memory and are derived from encountering similar situations as those from past experiences. However, because the deployment of ADAS in vehicles has been recent, drivers may lack accurate mental models about ADAS given the minimal experience, as well as lack knowledge about system functions and limitations as noted earlier. Incomplete knowledge or mismatched expectations about a system's function and limitations may result in lack of mode awareness hindering its user's ability to allocate appropriate attentional resources and detect errors, failures, and miscommunications between the user and the system (Sarter & Woods, 1995). In the driving domain, such user-related errors have been known to play a critical role in motor vehicle crashes.Singh (2015) found that driver related-errors were assigned as the critical reason in about 94% of crashes, where the term ‘critical reason’ was defined as the last event in the crash causal chain. It was also found that about 41% of the crashes were caused by recognition-related errors, while decision-based errors and performance-related errors were the cause for 33% and 11% of the crashes, respectively. This inaccuracy or incompleteness of mental models could lead to mode confusion (Wilson et al., 2020), miscalibrated trust with regards to system capabilities (Beggiato & Krems, 2013; Kidd et al., 2017), and may result in operator errors during ADAS usage (Pradhan et al., 2020; Pradhan et al., 2021; Stanton & Salmon, 2009). Hence, improving drivers' mental models is a critical requirement for appropriate and safe use of vehicle technologies.

Driver's mental models could be improved through driver training. Driver education has helped in improving skills related to hazard perception, visual search, and situational awareness (Horswill et al., 2015; Vlakveld, 2014; Walker et al., 2009). Driver training has also improved drivers' cognitive driving abilities (Yamani et al., 2016), which could indicate that training and education of drivers can be used to improve novice drivers' knowledge and awareness of advanced driver assistance systems. Training can be defined as ‘a planned and systematic effort to modify or develop knowledge, skills and attitudes through learning experiences, to achieve effective performance in an activity or a range of activities’. Learning can be defined as ‘a relatively permanent change in behavior or in the behavioral potential that results from experience’ (Garavan, 1997). Learning can therefore be understood as one of the outcomes of training. Learning also consists of many aspects, such as learning environment, abilities and learning style (Koć-Januchta et al., 2017; Stern, 2017). Learning style in this case can be thought of as an individual's preferred way of learning (Plass et al., 1998). Learning styles and preferences are usually based on four major methods - visual, auditory, kinesthetic and tactile (Klašnja-Milićević et al., 2016). According to previous studies, majority of the population are visual learners (Zopf et al., 2004) and learn using pictures, videos, etc. Studies have also found that when learners were instructed to form images while reading texts or received pictorial cues, their recall accuracy and retention was high (Paivio, 2014; Sadoski & Willson, 2006; Tabbers et al., 2004).

While training leads to learning and skill development, one could seek knowledge and experiences of their own volition, which could also have similar learning outcomes and skill development. The focus of this study is to improve driver knowledge solely through training methods. The impacts of driver training have been widely examined in the literature, and there is strong evidence of improvements after training in driver behaviors such as attention maintenance (Pradhan et al., 2011), hazard perception (Pradhan et al., 2005), and hazard mitigation (Muttart, 2013). However, these training approaches have been studied and evaluated for the traditional driving domain. Since these training approaches were not designed for or targeted any vehicle technologies, there is little that can be extrapolated from them in terms of impact on mental models of advanced technologies.

The literature on driver training in the domain of vehicle automation is somewhat sparse. Some prior research has shown training to be effective in improving driver's knowledge about system limitations, but the results have been somewhat mixed. In a study where the training material provided was weak or strong (given via powerpoint presentations), it was found that drivers with strong mental models were better at operating ADAS during edge-case situations than those with weak mental models (Gaspar et al., 2020). Another study showed that user education through owner's manual and interactive tutorial led to increased understanding of driving automation systems (Forster et al., 2019). In this study, participants were given either baseline information (generic information about L3 and L2 systems) or Owner's Manual (information delivered in a four-page, short sentences and bulleted format, taken from a BMW manual) or interactive (participants completed a tutorial and answered questions).

In contrast, another study suggested that training (provided as descriptions of the ACC interface and explanations of icons displayed) improved drivers' abilities to detect system notifications and change in system status, but only moderately improved drivers' comprehension of system limitations (Mueller et al., 2020). Victor et al. (2018) also reported that of drivers who received specific instruction about vehicle limitations, as well as supervision reminders to keep eye on road and hands on wheel, 28% still collided with a conflict object in their experimental field study. While a large majority of the participants (72%) did benefit from the supervision and instruction (training), it is still critical to examine the failures, and the authors acknowledge the need for more research to examine how to communicate system limitations to drivers. Similarly, Noble et al. (2019) reported that training strategies (baseline and interactive module where participants watched videos and interacted with the system as instructed) only led to limited differences in driver knowledge and no difference in driver behaviors or attitudes.

Drivers' knowledge can contribute to their trust in a system, and thus to the appropriate use of a safety system. However, while there is much work conducted in this domain, there is still mixed evidence about the impact of training on driver trust. In study conducted by Payre et al. (2017), simple or elaborate (text and videos) training optimized drivers' trust when driving with automated features and decreased time to respond to emergency situations. Similarly, Koustanaï et al. (2012) familiarized drivers with Forward Collision Warning system (an ADAS feature) through a simulator-based training and also found improvement in trust towards the system and improvement in driver-system interactions. Three other studies (Beggiato & Krems, 2013; Kazi et al., 2017; Lee & See, 2004) also found that the correct use and knowledge of ACC depends on the level of trust in the system. However, when comparing limitation-focused training and responsibility-focused training approaches,DeGuzman and Donmez (2022) found that there were no differences between the two approaches regarding the drivers' ADAS related knowledge, but found that both approaches negatively affected trust for scenarios where ADAS may not work. Similarly, Zahabi et al. (2021) found no significant differences in driver trust and knowledge about automation when comparing between demonstration-based and video-based training programs.

Overall, we need more evidence about the impact of training on improving drivers' mental models. Traditionally, educating drivers about vehicle capabilities and features have been conducted using Owner's Manual, which has been described as tedious and too complicated (Mehlenbacher et al., 2002). In this study, we compare multiple training methods, a Text-Based training method which is based on the Owner's Manual and a Visualization method which describes the different states of Adaptive Cruise Control using state diagrams. We examine the effect of training method and content to understand the impact of different approaches on drivers' mental models.

Study objective and hypotheses

The objective of this study was to understand the impact of different types of training approaches and content on improving drivers' mental models of Adaptive Cruise Control (ACC), and to understand if that is related to driver trust.

This objective has been motivated by mounting evidence in the field that training and education of drivers may help in improving their understanding of Advanced Driver Assistance Systems (Forster et al., 2019; Gaspar et al., 2020). In addition, there is a rich field of research that underlines the utility of visual learning in improving recall and retention of information (Paivio, 2014; Sadoski & Willson, 2006; Tabbers et al., 2004). Finally, given the importance of user trust in appropriate use of technology, it is important to understand if trust may be related to the depth of one's understanding of a system. There is evidence that trust in advanced drivers' assistance systems may be related to drivers' mental models (Beggiato & Krems, 2013; Kazi et al., 2017; Lee & See, 2004). However, we need more evidence on the benefits of training, including evaluation of multiple types of training and the relationship to trust.

Given these objectives and motivations, this experimental driving simulator study was conducted to test the following hypotheses:

• Training improves drivers' mental models of ACC (as measured by knowledge).

• Training that includes visualization will be more effective than those without.

• Drivers' mental model improvement will correspond to drivers' trust in ACC.

Participants

36 participants (M = 31.58; SD = 10.23; min = 21; max = 53; Female = 22) were recruited for this study. Drivers with valid US driver's licenses and naïve to ACC were eligible for participation. On average, participants had been licensed for 12 years (Min = 1; Max = 39) and drove for 50–100 miles weekly. A participant's familiarity with ACC was established by a series of screening questions about their prior experience and familiarity using ACC. Only those participants who self-reported as being either novice users or having no knowledge about ACC were included in the study. The study sessions were conducted online through the Zoom video- conferencing platform. Institutional Review Board approval was granted for conducting the study.

Experimental design

The study was conducted as a between-subject experiment with Training Method as the independent variable and participants' system knowledge as the dependent variable. This section details the experimental design of the study.

Experimental procedure

The experimental session was conducted online on the Zoom video-conferencing platform and the session lasted for approximately an hour. Participants were randomly allocated to one of the three groups immediately after their study session was scheduled. This randomization was done by the experimenter prior to any assessment using a predefined randomization table. Blinding was not used because the training interventions were presented by the experimenter, and the group assigned were based on the type of training intervention the participants received. Each of the sessions began with the participants being informed and consented for participation. The experimenters then gave them a brief overview about the study session. Under the supervision of the experimenters, participants accessed the different surveys through individual links provided by the experimenters. Participants first completed the pre-training surveys covering general demographic information and a Trust Survey (Jian et al., 2000). Following these, the participants received the Mental Model Survey. The participants then completed the training material based on their group allocation. The participants received a link to the training material which led to a document the participants read through. Following the training, participants completed the post-training Mental Model Survey and Trust Survey. After completing the post training surveys, particuipants were paid and the session was concluded. The entire session from beginning of the session to the end took around 60 - 70 minutes.

Mental Models Survey scores

The post-training Overall Mental Model Survey scores were analyzed across the three training groups using ANCOVA (analysis of covariance) where the pre-training overall scores were treated as the covariate. The covariate, pre-training Overall Mental Model Survey score, was significantly related to post-training Overall Mental Model Survey scores (F (1, 32) = 23.412; p = 0.00003; η2 = 0.422). The ANCOVA found that there was no main effect of the training method on the post-training scores after adjustment for the pre-training scores (F (2, 32) = 2.208; p = 0.126; η2 = 0.121). Figure 4 shows the pre-training and post-training mean Overall Mental Model Survey scores across all three training groups.

Additional analyses were conducted to analyze the differences between the groups in terms of the completeness and accuracy scores. An ANOVA found a main effect of both survey time (F (1, 32) = 45.408; p < 0.00001), and training method (F (2, 32) = 4.276; p < 0.0226) for the completeness only scores, and a main effect of survey time (F (1, 32) = 13.973; p = 0.0007) for the accuracy only scores (Figure 6; Figure 5).

Correlation between Overall Mental Model Survey and Trust scores

Previous work has reported that training did not have any impact on the user's post-training overall trust scores (Pai et al., 2021). However, it is unknown if there is a relationship between the mental models scores and overall trust scores. A Pearson's product-moment correlation test was conducted to examine the correlation between the overall mental model scores and the overall trust scores. The correlation test revealed no significant correlation between the pre- training Overall Mental Model Survey scores and the pre-training overall trust scores, r (34) = 0.022; p = 0.898. The test also revealed no significant positive correlation between the post- training Overall Mental Model Survey scores and the post-training overall trust scores, r (34) = 0.254; p = 0.135. (Figure 7).