Numerous concurring definitions of cycling comfort can be found in literature (CROW, 2016; Castañon & Ribeiro, 2021; Landis et al., 1997; Yamanaka & Namerikawa, 2007; Yamanaka et al., 2013). However, there are certain recurring aspects of comfort mentioned by the majority of works on the topic. Both cycling comfort in general and its sub-aspect surface roughness are often referred to in so called bikeability indices trying to assess how well-suited a given area is for cycling. The review of bikeability indices by Castañon and Ribeiro (2021) provides a good overview of recent works on the topic and, more relevant here, bikeability criteria considered by these works. The criteria analyzed most often are geometric design features of the cycling infrastructure, the accessibility of relevant areas, the traffic safety and the topography of the study area. Each of these aspects was considered by eight out of the 14 works they looked at. The geometric design features include the surface type and condition, used by 5 out of the 14 works as a bikeability indicator. The only other indicator also used by five of the included works was the presence of dedicated cycling infrastructure. Thus, the surface roughness, considered to result from the combination of surface type and condition, can be considered to be among the most relevant aspects of cycling comfort. (Castañon & Ribeiro, 2021)

Yamanaka et al. (2013) and Yamanaka and Namerikawa (2007) developed and calculated a bicycle level-of-service in two separate works. In the first work (Yamanaka & Namerikawa, 2007), they listed the roughness of the road surface, the required effort by the cyclist, and braking, slow speed, speed deviation and comfort of speed as the main factors of a bicycle level-of-service. For the second work (Yamanaka et al., 2013), they changed this list to the stability of the speed, stops, vibrations, steering, traffic density, and braking behavior. Although in the later work vibrations were considered less important, they still confirm surface roughness to be a relevant aspect of cycling comfort.

Surface roughness being of major importance to cycling comfort is not a new concept either. In the late eighties, Axhausen et al. already found it to be of importance especially for experienced cyclists in a preference study (Axhausen & Smith, 1986). This was later experimentally proven by Landis et al., finding a strong correlation between pavement surface condition and participants comfort perception (Landis et al., 1997). Last, the English version of the Dutch CROW Design Manual for Bicycle infrastructure, often considered a kind of gold standard for cycling infrastructure, states ‘The surfacing on the road section should satisfy the requirements in terms of evenness.’ (CROW, 2016).

Measuring surface roughness of cycling infrastructure based on accelerations is a well established approach with recurring components found in literature. While several, especially earlier, works used dedicated accelerometers (Bíl et al., 2015; Gao et al., 2018; Neto et al., 2018; Olieman et al., 2012), most recent works rely on smartphones for data collection (Kranzinger & Leitinger, 2021; Litzenberger et al., 2018; Nuñez et al., 2020; Wijerathne et al., 2018; Zang et al., 2018). In most cases, authors use vertical acceleration values, either by installing the IMU in such a way that one axis is perpendicular to the ground plane (Bíl et al., 2015; Nuñez et al., 2020), or by calculating the resulting vertical acceleration from all three axes (Kranzinger & Leitinger, 2021; Neto et al., 2018; Zang et al., 2018). Some works use all three axes of the accelerometer directly (Gao et al., 2018; Litzenberger et al., 2018). The accelerometers and smartphones in these studies are mounted to the fork (Bíl et al., 2015), the handlebar stem (Kranzinger & Leitinger, 2021; Zang et al., 2018), the left or right half of the handlebar (Gao et al., 2018), or the front (Litzenberger et al., 2018; Neto et al., 2018) or back (Nuñez et al., 2020) of the top tube. Typical sampling frequencies for the acceleration data range between 20 Hz (Bíl et al., 2015) and 100 Hz (Litzenberger et al., 2018; Zang et al., 2018), with 50 Hz being most common (Harikrishnan & Gopi, 2017; Kranzinger & Leitinger, 2021; Neto et al., 2018; Nuñez et al., 2020; Wijerathne et al., 2018) and occasional higher values up to 512 Hz (Calvey et al., 2015).

With regard to the roughness calculation, ISO 2631-1 (ISO, 1997) on whole body vibrations is often mentioned as a basis for roughness measures and indices (Gao et al., 2018; Nuñez et al., 2020). Furthermore, the IRI (Sayers & Karamihas, 1998) is a well-established roughness measure. Therefore, some works try to directly calculate it (Zang et al., 2018), while some consider closely related measures (Bíl et al., 2015; Gao et al., 2018). Other works introduce their own comfort measures not related to ISO 2631-1 or the IRI (Kranzinger & Leitinger, 2021; Wijerathne et al., 2018).

Roughness calculation is typically performed on a segment-wise basis. There is no standard segment length, with 5 m (Nuñez et al., 2020), 10 m (Kranzinger & Leitinger, 2021), 20 m (Zang et al., 2018), and 100 m (Bíl et al., 2015) all found in literature. In some works, test rides are not segmented further and instead roughness is calculated per test route. Gao et al. (2018) analyzed test routes with an average length of 250 m, while (Litzenberger et al., 2018) choose test routes with approximately 100 m each. Neto et al. (2018) did not group the measured accelerations by travel distance but by time, calculating roughness for every 2 seconds of collected data.

In recent years, sensor equipped bicycles have become a lot more prevalent than before, also yielding more works on surface roughness assessment. A very good overview of the use of these sensor bicycles in research was compiled by (Gadsby & Watkins, 2020). They list six works investigating pavement condition. Two works by the same research group (Calvey et al., 2015; Taylor & Fairfield, 2010) describe a sensor bike suited for acceleration measurement but no actual calculation of surface roughness. One work used a neural network to classify different pavements (Neto et al., 2018), however their description was considered unsuited for reimplementation. Two works (Bíl et al., 2015; Nuñez et al., 2020) were chosen for comparison and are described in the next section.

Wijerathne et al. (2018) describe a sophisticated approach to calculate surface roughness independent of accelerometer position and orientation. To this end, they used four smartphones, one mounted on the handlebar, one in a front bag, one attached to the test riders arm, and one carried in a backpack. They used a wavelet-transform followed by a Taylor expansion and tested for Lipschitz continuity to detect bumps. Although impressive, their work was not chosen for comparison in this work. The reason is their focus on detecting single bumps, making comparison with methods yielding a segment-wise roughness metric difficult (Wijerathne et al., 2018).

From the numerous works using IMUs on bicycles for surface roughness calculation, a selection was made based on three considerations:

Ultimately, five works were chosen for comparison. The works by Bíl et al. (2015) and Zang et al. (2018) are the most cited works on roughness calculation from accelerations measured using a sensor-equipped bicycle. Bíl et al. (2015) describe their algorithm for roughness calculation commendably concise. Zang et al. (2018) on the other hand are among the few authors using more than one accelerometer axis for their calculations. However, as a first step they derive the vertical acceleration for further processing. Gao et al. (2018) are among the even fewer authors actually using all axes and their work is still fairly popular. All of these methods produce numerical metrics. However, for further use in categorization or maintenance of bicycle infrastructure, ordinal metrics are often preferable. Therefore, two works presenting methods yielding such metrics were also chosen for comparison: The work by Nuñez et al. (2020) combines video analysis and acceleration measurements for assessing the quality of cycling infrastructure. To this end, they also present an adaption of the ISO 2631-1 (ISO, 1997), yielding six comfort classes for surface roughness. Last, the method presented by Kranzinger and Leitinger (2021) has been chosen for comparison. It also yields an ordinal metric for surface roughness and, more interestingly, is not based on the acceleration itself but its first derivative, the so called jerks. The methods for roughness calculation from accelerations used in these works are described in detail in section 3, together with descriptions of the corresponding reimplementations. The remainder of this chapter outlines the general approach of each of these works, especially aspects not found in other works.

Bíl et al. (2015) measure accelerations using a relatively low frequency of 20 Hz, compared to the 50Hz found in most works. The accelerometer is attached to the fork of the bicycle in such a way that one axis is considered vertical and used for roughness calculation. The separate GNSS sensor is mounted to the handlebar, both signals are combined after collection based on their timestamps. For roughness calculation, only downwards accelerations >1 g are used. The resulting so-called dynamic comfort index (DCI) ranges between zero and one. Higher values correspond to more comfortable roads. (Bíl et al., 2015)

The work by Zang et al. (2018) is based on the idea of the quarter-car model used for calculation of the IRI (Sayers & Karamihas, 1998). They measure accelerations at 100 Hz and GNSS localizations at 1 Hz using a smartphone mounted to the handlebar stem. Vertical acceleration is derived by projection of measured accelerations with a reference unit vector calculated from stand still points at the beginning of each test run. The IRI is meant to represent vertical displacement. In order to calculate it, double integration of the vertical accelerations is used. The resulting IRI is measured in mm/m vertical displacement and ranges from zero to approximately 12 mm/m on their test tracks. (Zang et al., 2018)

Gao et al. (2018) used an accelerometer mounted on the left half of the handlebar and a separate GNSS sensor for data collection. They report neither acceleration nor GNSS recording frequency. Their test subject was instructed to keep a constant speed between 12 km/h to 16 km/h. Following ISO 2631-1 (ISO, 1997), they weighted the measured accelerations in the frequency domain using a small band filter. Unfortunately, they do not report on the parameters of the weighting or the filter. Notably, they use all three axes of the accelerometer for roughness calculation without extracting the vertical component first. Their so called Dynamic Cycling Comfort (DCC) ranges between zero and just above three for their test routes. (Gao et al., 2018)

Nuñez et al. (2020) use two sensors to calculate their so called Bicycle Environment Quality Index (BEQI), a camera mounted at the front of the frame for capturing videos and a smartphone mounted on the top tube in front of the saddle to capture accelerations. The smartphone is mounted parallel to the ground such that a single accelerometer axis is considered the vertical acceleration. Accelerations are sampled at 50 Hz. Notably, they used frequency analysis to find characteristic frequencies of different pavement types. Based on ISO 2631-1 (ISO, 1997), roughness is rated as one of six classes depending on the root mean square of vertical accelerations. (Nuñez et al., 2020)

Kranzinger and Leitinger (2021) are the only authors using the derivative of vertical accelerations, called jerks, to calculate surface roughness. They assessed the surface roughness of impressive 436.6 km of cycling infrastructure, analyzing 5 166 km of test runs. Accelerations are captured using a smartphone mounted to the stem of the handlebar. Accelerations are sampled at 50 Hz and GNSS localizations at 1 Hz. Subsequently, four roughness classes are calculated per 10 m segment using the k-means++ clustering algorithm (Arthur & Vassilvitskii, 2007). As input parameters, the average jerk value and the share of jerks above a given threshold of 1 200 m/s³ are used. The roughness metric of each section is then defined as the class it is assigned to by the clustering. (Kranzinger & Leitinger, 2021)

The data used in this work was collected between September and November 2023 in four cities and villages in Austria. The test routes are depicted in Figure 3. Each route was driven 8–16 times. A smartphone and a dedicated IMU-GNSS-device were mounted side by side for simultaneous data collection. For each route, data was collected in one continuous session, without remounting of any sensors, as to keep their position and orientation steady. The collected data was matched to the so called ‘Graphenintegrationsplattform’ (GIP), a national Austrian digital road graph (Neto et al., 2018; ÖVDAT, 2022). The generated trajectories were split into segments of differing length for segment-wise roughness calculation. Data from both sensors was brought to a common data format and collected in a single data set for further processing.

To calculate the roughness based on the accelerations grouped by segments there are some preprocessing steps followed by the actual roughness calculation methods. First, required for all compared methods but the one by Gao et al. (2018), the vertical component needs to be extracted from the three accelerometer axes. Also accelerations are filtered based on speed to clip stop points, slow sections and also GNSS errors resulting in very high speeds and possibly erroneously map matched measurements. Finally, the actual roughness calculation using the reimplementations of the methods selected for comparison is performed. The result is one rating per test segment, sensor and method to be compared subsequently.

The stated goal of this work as outlined in section 1 is to compare different sensor-method combinations and segment lengths used for surface roughness estimation. To this end, mathematically sound comparison methods are required. While numerical assessments are fairly easy to compare assuming that the assessments are normalized first, the mixture of numerical and ordinal assessment scales proved difficult to compare. For stability, no possibility of comparing numerical and ordinal assessment methods directly was found. Comparing the numerical assessments to each other was straightforward using the coefficient of variation. Comparing ordinal assessments is difficult, as one must not make the mistake of considering classes equally spaced. However, a suited measure for ordinal dispersion was found that allows for comparison of stability between ordinal methods and their combination with different sensors and segment lengths. Regarding the reliability, defined as the agreement of any given method with an ensemble of all considered methods, rank correlation was used to be able to compare all used methods to each other.

As pointed out in section 3 the stability of numerical and ordinal assessment methods are not directly comparable. As to the best of the authors knowledge, no dispersion metric applicable to both types of scales exists. However, to portray the effects of sensor and segment length, they are still displayed side by side in Figure 5. At the top of each of the four box plots, the method by Kranzinger and Leitinger (2021), denoted ‘Kranz.’, and the BEQI (Nuñez et al., 2020), the two ordinal methods, are shown. As described, the other three methods yield numerical assessments. As can be seen, the used method has the biggest impact on dispersion, consistent across sensor type and segment length.

Of the ordinal methods, the BEQI is more stable and its dispersion is less impacted by sensor type than the assessment using the method by Kranzinger and Leitinger. Notably, the method by Kranzinger and Leitinger appears to be the least impacted by chosen segment length. Of the numerical methods, the DCI shows exceptionally stable results across both sensors and all segment lengths. The IRI, calculated as described by Zang et al. (2018) shows a very high dispersion, indicating highly unstable results across test runs. The DCC is somewhat in between, without particularly stable or unstable results. However, among the numerical methods it is least impacted by segment length.

Next, the segment length has a major influence on dispersion. Unsurprisingly, longer segments tend to yield more stable results, except for the IRI calculation as described by Zang et al. (2018). This was to be expected, as the longer the section is, the smaller is the impact of little differences in the actually driven path. For short segments, even single swerves hitting or avoiding any unevenness have a major impact. This is a disadvantage of any IMU-based method: only the surface directly beneath the tires can be assessed. Notably, as there are about 20 times less segments with a length of 100 m than with a length of 5 m, there are also a lot less outliers for longer segments.

For shorter segments, the assessments made using the XSens are in general less stable than the ones using a smartphone, with the exception of the combination with the DCI. This also indicates a bigger dispersion in the measured accelerations themselves. However, for longer segments, this effect disappears, likely due to the described ‘smoothing’ effect of adding more data points.

In conclusion, the most stable numerical assessments are achievable using the combination of dedicated industrial-grade IMU and the DCI method by Bíl et al. (2015). However, especially when using the DCI, the effect of the used sensor is neglectable. The least stable assessments are produced when calculating the IRI as described by Zang et al. (2018) based on measurements by the dedicated IMU. Of the ordinal methods, the BEQI is more stable, especially for shorter segments and when using the dedicated IMU for acceleration measurement.

As described in section 3 the reliability of each method is defined as its agreement with the other methods, as well as an assessment averaged from all considered methods. Rank correlation is used to be able to properly compare numerical and ordinal assessment methods. All sensor-method combinations are correlated against each other as well as against the average assessment calculated as described in section 3. The correlations are depicted for the different segment lengths in Figure 6 to Figure 9. The resulting reliability metric ranges between 1 for total accordance, as seen in the correlation of each combination with itself; and -1 for exact opposite ranking of sections.

Notably, the DCI (Bíl et al., 2015), DCC (Gao et al., 2018), and BEQI (Nuñez et al., 2020) are all highly correlated. This is a result of them all being based on the root mean square of accelerations. The DCI only uses downward accelerations > 1 g, the DCC takes accelerations from all accelerometer axes and the BEQI projects the results onto an ordinal scale. But beyond that, they all use a root mean square of accelerations for vibration assessment as recommended by ISO 2631-1 (ISO, 1997).

This yields a certain problem for taking the correlation with the averaged assessment as a reliability measure, as these three methods might appear disproportionately reliable. For future applications of this comparison approach, this might be alleviated by a more diverse selection of methods. For this work, the presented correlations need to be interpreted with due care.

Concerning the effect of segment length, longer segments yield a generally higher correlation of different methods. Especially between the 5 m segments and the 10 m segments, correlation improved for the majority of pairwise compared sensor-method combinations. Increasing the length further yielded small gains for 20 m segments, and somewhat inconclusive results for 100 m segments. Contrastingly, the correlation of the IRI, calculated as described by Zang et al. (2018), decreased with segment length. This indicates some kind of difficulty this method encounters with longer segments.

As with stability, the impact of the two different sensors decreases with segment length. Also, it is again highly dependent on the used roughness calculation method. The three aforementioned, root mean square based, methods show high correlations with themselves and towards each other for both sensors used. The IRI ranking is less correlated to the IRI ranking using the other sensor, however still more than to any other method. In general, the IRI is actually negatively correlated to all other methods, indicating some major problems of either the method or the used reimplementation.

The method by Kranzinger and Leitinger is most interestingly stronger correlated to other methods than to itself using another sensor, especially for longer segments. Its overall correlation to other methods is rather weak, although increasing with segment length. However, as mentioned three of the methods are based on the same fundamental concept of a root mean square of accelerations. The method by Kranzinger and Leitinger on the other hand is based on a clustering by mean jerks and share of heavy jerks. Jerks are defined as the derivative of the measured vertical acceleration. Presumably, this approaches' sensibility to few very high accelerations is different from the root mean square based approaches. Therefore, comparison to another method considered sound in itself would be required to achieve a trustworthy estimation of its actual reliability for surface roughness assessment. Interestingly, this method is hardly correlated to itself using a different sensor. Presumably, this is the result of the different heavy jerk thresholds and cluster centers calculated for the two sensors.

As a minimal qualitative example of results, Table 2 lists the ranks of all methods for the on average best and worst assessed segments per segment length and used sensor. As can be seen, the agreement of methods for XSens measurements on these segments is generally high, especially for the good sections. Interestingly, the agreement decreases with segment length. As with stability and reliability, the IRI does not fare too well, again worse for longer segments. The ranks of the two ordinal methods in the last two columns are expected to be less clear, as for all segments in one class, an average rank is used. This also yields the 0.5 ranks.

For accelerations measured using a smartphone, there is less of an agreement, especially for the good segments. Interestingly, for the good sections, the method by Kranzinger and Leitinger appears to yield better results, however this might also be a misinterpreted effect of the class distribution of the two ordinal methods. Furthermore, the best segments all feature asphalt in good condition, while the worst sections all feature rough cobblestone surfaces. This confirms the general assessments of the used methods. However, 16 out of about 4 000 segments can at best be used to get a general idea of the characteristics of each method. For proper analysis of the methods, quantitative approaches as described are required.

The described reliability, stability, and exemplary qualitative results allow for some deductions. First and foremost, the selected method has the biggest impact on the resulting assessment, regarding both reliability and stability, well above the impacts of used sensor and segment length. This supports the ongoing research into better assessment algorithms. Furthermore, it sets the focus for any attempts at assessing infrastructure surface quality in a given area.

Second, segment length had a bigger impact on reliability than on stability. This indicates that using several measuring runs is a well suited approach to stabilize the outputs of the considered methods. As a side effect, the impact of the different sensors was less visible on reliability than on stability. However, it still mostly depended on the used method. Overall, except for the IRI, longer segments yield more similar assessments both per test run and also when comparing methods. On the other hand, they also decrease spatial resolution and might impede the detection of small problematic areas by a ‘smoothing’ effect.

Furthermore, the effect of different sensors, although dependent on segment length, can be considered to have a minor impact on the resulting assessment. Especially for the three root mean square based methods, results using either sensor were very comparable. For the IRI, the generally bad results are worse when based on smartphone measurements. Regarding the method by Kranzinger and Leitinger, it is interesting to note that there is little correlation when applying the method to the acceleration data collected using either sensor. However, as noted it is the only method with parameters tuned to each sensors measurements.

Last, the coherent results of both the stability and reliability measures indicate the chosen comparison methods to be well suited for the task at hand. This supports the assumptions leading to their selection and might be a useful finding for any future works aiming to compare their own, newly developed, methods to existing ones. As the described methods are mathematically well defined and straightforward to implement, little effort is needed to apply them in any such future works.