Model name: Crank validity model

Goal: The objective of the model is to use the features of the cranking voltage profile, derived from the time series voltage data sent by GO devices, to determine whether a cranking event is valid or invalid.

Base model: Gradient boosting classifier

Model type: Supervised, binary classification

Model version: 14

Developed by: Vehicle maintenance analytics team

Primary intended uses:

• The model is used to classify cranking events as valid or invalid logged by GO7/8/9 devices on firmware version 28 or newer.
• The data from the valid cranking event can be used in predictive maintenance requirements. For instance, the minimum voltage during cranking is used to determine the health of the vehicle's electrical system.

Out-of-scope uses: The following devices do not transmit voltage data during an event or do not provide sufficient data quality for accurate predictions. Therefore, the model should not be used to classify cranking events from the following devices:

• GO6 devices
• GO7/8/9 devices on firmware version older than 28
• Devices from original equipment manufacturer (OEM)
• General third party devices

Targeted users/user groups: Users who are interested in using the voltage profile of cranking events, especially for predictive maintenance, to identify potential battery or electrical system failures can leverage the model's outcome to filter erroneous data.

Factors impacting model performance: The accuracy of the model varies depending on the vehicle type (make, model, fuel type). This is based on an evaluation of the test dataset, which showed that some vehicle types have lower accuracy than others. These categorical features are not included in the model.

Description on training and test data

• The GO device logs the data, and the raw GPS data is used to identify ignition events. These events then serve as an anchor to align the cranking voltage data records to an event.
• The model uses features that describe the voltage profile during a cranking event. These features include voltage recordings, voltage differences, specific voltage points (such as first and second maximum), total event duration, and the duration of the actual cranking within the event.
• To maintain a consistent distribution across vehicle types, stratified sampling is utilized to create a balanced dataset. This dataset is then divided equally into training, validation, and testing sets, preserving the uniform distribution.

Data pipeline and preprocessing

• The data pipeline is regionalized in alignment with the data residency standards (6 regions). It runs daily and classifies all cranking events on the previous day as valid or invalid using the Airflow orchestration (directed sequence of tasks with no loops or cycles).
• The pipeline cleanses raw data, and downstream applications make use of features of the valid cranking events.
• Data processing steps:
  • Cranking events typically last less than 10 seconds. The time between cranking voltage events that are greater than 10 seconds apart indicates a new cranking event.
  • Ignition on events are determined using GPS records and time since last cranking voltage.
  • Once all the events are identified, and all cranking voltage records are assigned to an event, the features for the cranking event voltage profile are calculated.
  • The aforementioned features are then fed into the machine learning model to perform the classification.

Data privacy

• There is no Personally Identifiable Information (PII) utilized during training, evaluation, or serving.
• Inputs to the model are simply numerical features representing characteristics of the cranking event voltage profile.

In this section, we highlight some ethical challenges that were encountered during the model development, including bias and fairness considerations, and present our solutions to overcome these challenges. Additionally, we provide the assumptions and constraints of our model, including any limitations in the data or the model’s scope that could affect its performance, in order to foster the understanding of the model's strengths and limitations to the stakeholders which is crucial to use the model responsibly and interpreting its results.

Risks in training

• The ML model uses numerical features for training. It does not include some relevant categorical features, such as vehicle Make, Model, and Fuel Type.
• The training and testing data exhibit an imbalance in vehicle types; for instance, there are significantly more samples for a Ford Transit Gasoline than for a Volkswagen Golf Gasoline.
• This imbalance can lead the model to learn patterns more effectively for some vehicle types while underrepresenting others.

Data bias handling

• To mitigate the aforementioned risk, the distribution of samples per vehicle type were analyzed, minimum and maximum sample thresholds were set for the training, validation, and testing datasets.
• This methodology ensures a comparable quantity of samples for each vehicle type, while maintaining a sufficiently large dataset for both training and evaluation.

Model assumptions and constraints

The model assumes that Make, Model, and Fuel Type are the key categorical features. However, if another relevant categorical feature—such as Year—should also be considered, its exclusion may have led to an inaccurate determination of the number of samples per group in the training and evaluation datasets.

The model was evaluated on the unseen test data and the performance metrics used to assess the model are accuracy, precision and recall.

• Accuracy measures the overall percentage of correct classifications. The best model has an accuracy of 99.6%, meaning that it correctly predicts whether a cranking event is valid or invalid 99.6% of the time.
• Precision measures how often positive predictions are accurate. The precision of the best model is 99.5%. This means that when the model predicts an event as valid, it is correct 99.5% of the time.
• Recall quantifies how well the model classifies the actual positive cases. The recall of the best model is 99.7% meaning there is only a 0.3% chance that an event will be missed as a valid cranking event.