Model Card: Event Reviewer

Model name: Event Reviewer

Goal: The primary goal of the Event Reviewer is to automatically review uploaded event data (short video clips and auxiliary sensor data) to determine if the initial dangerous event recognition was a true positive or a false positive. This is crucial for filtering out false positive events so they are not shown to the customer and are excluded from the calculation of the driver safety score.

Developed by: GVP Computer Vision Team

Primary Intended Uses:

• Automated binary classification of dangerous event triggers (identified by the AI dashcam) as either True Positive (a genuine dangerous event) or False Positive (an incorrect identification).
• Filtering the stream of recognized events to improve data quality, ensuring only true positive events are presented to the customer and used for driver safety scoring.
• Providing annotations on the event to facilitate downstream analysis or human review processes.

Out-of-scope Uses:

• Real-time driver intervention or in-vehicle alerting based on the event review decision.
• Detailed root cause analysis or investigation of true positive events.
• Generating comprehensive accident reports or legal documentation.

Target Users/User Groups:

• Fleet Managers / Safety Officers (to ensure accurate safety scoring).
• Internal Data Science/Engineering teams (for model monitoring and data quality assurance).

Description on Data and Features

The model utilizes multimodal data captured by an AI dashcam. The input data for an event review consists of a short video clip and auxiliary sensor data (e.g., speed, accelerometer readings, GPS) recorded and uploaded to the cloud following the initial dangerous event recognition.

Data Pipeline and Preprocessing

Data Ingestion and Source

Data is sourced from AI dashcams installed in vehicles. The ingestion process is triggered when the dashcam's on-device model recognizes a dangerous event (on-road or in-cabin). Upon recognition, a short video clip and associated auxiliary sensor data (speed, accelerometer readings, GPS) are recorded and uploaded to a secure cloud platform for processing by the Event Reviewer model.

Data Preprocessing for Model Training

The multimodal input requires distinct and synchronized preprocessing streams:

• Video Processing: Short video clips are standardized, which may include transcoding to a consistent format, resizing frames (e.g., to 224x224 pixels) for model input efficiency, and normalizing pixel values across all channels. Temporal synchronization with sensor data is critical.
• Auxiliary Sensor Data Processing: Time-series data (speed, accelerometer readings, GPS) is sampled, synchronized to the video timeline, and normalized to ensure consistent scale and distribution across features. This may also involve smoothing or filtering to reduce noise.
• Labeling and Ground Truth Generation: The raw events, which are initial recognitions by the on-device model, are reviewed (potentially by human annotators) to assign the definitive ground truth label: True Positive or False Positive. This high-quality, verified binary label is essential for supervised model training.
• Data Quality and Filtering: Incomplete, corrupted, or extremely short event clips and associated sensor logs are identified and excluded from the training set to maintain data integrity.

Model Training and Storage

The Event Reviewer is based on a multimodal deep learning architecture, typically a specialized neural network designed to integrate features from both the processed video stream and the auxiliary sensor time-series data.

• Training Method: The model is trained using a supervised learning approach on the high-quality, ground-truth labeled events (True Positive or False Positive) generated during the data preprocessing stage.
• Optimization: Training optimizes a binary classification loss function to maximize the classification accuracy, precision, and recall on the True Positive class, with a focus on minimizing the False Negative rate.
• Hardware and Infrastructure: Model training is performed on a secure cloud-based infrastructure utilizing GPU accelerators.
• Model Artifact Storage: The trained model's weights, configuration, and version metadata are stored in a secure, version-controlled Model Registry. This storage ensures traceability and facilitates seamless deployment and rollback procedures.

Post-processing for Forecast Output

The model generates a raw output, typically a probability score indicating the likelihood of the event being a True Positive. This score is then processed to create the final, actionable output:

• Threshold Application: A predetermined classification threshold (e.g., 0.5 or an optimized value) is applied to the raw probability score to produce the definitive binary classification: True Positive or False Positive.
• Event Filtering: The resulting False Positive events are flagged for immediate exclusion from the customer-facing system and are not used in driver safety score calculations.
• Annotation and Routing: The resulting True Positive events are annotated with the model's decision and confidence score. These high-quality, verified events are then routed to the customer platform for review by Fleet Managers/Safety Officers and used as input for safety scoring algorithms.
• Monitoring Data Generation: The final prediction, confidence score, and input features are logged to a secure database for ongoing model performance monitoring and data drift detection.

Data Privacy

Specific actions and considerations are implemented with a privacy-by-design approach to preserve the privacy and confidentiality of our customers and drivers. This includes:

• Data Minimization: Only necessary data (short video clips of the event and associated auxiliary sensor data) are collected and processed, reducing the overall volume of sensitive information.
• Anonymization and Pseudonymization: Wherever feasible and not detrimental to the model's objective, personal identifiers are removed or replaced with pseudonyms. This may include, but is not limited to, separating vehicle/driver IDs from the core event data during model training and storage.
• Secure Storage and Transmission: All video and sensor data is uploaded to and stored on a secure, encrypted cloud platform. Data transmission from the dashcam to the cloud is secured using industry-standard encryption protocols (e.g., TLS).
• Access Control: Strict role-based access control (RBAC) is enforced to ensure that only authorized personnel (e.g., internal teams for model monitoring, and authorized Fleet Managers/Safety Officers for reviewed events) can access the sensitive video and telematics data.
• Compliance: Processing aligns with relevant regional data protection regulations and company policies, particularly concerning the handling of personally identifiable information (PII) captured in the video footage.

In this section, we highlight some ethical challenges that were encountered during the model development.

Risks in Training

Training a multimodal model like the Event Reviewer presents several risks that must be actively managed to ensure a robust, fair, and reliable system:

• Training Data Distribution Shift/Incompleteness:
  • Risk: The training data may not fully represent the diversity of real-world driving conditions (e.g., weather, lighting, road types, camera mounting angles, vehicle types) and failure modes that could cause a dangerous event or a false trigger. A model trained on a limited distribution may perform poorly when deployed to new, unseen conditions.
  • Mitigation Focus: Continuous monitoring of the deployed model's prediction distribution against the training data distribution (data drift/concept drift detection). Active learning strategies to acquire and label data from underrepresented or new failure-mode scenarios.
• Overfitting to Specific Fleet/Driver Characteristics:
  • Risk: If a large portion of the training data comes from a small, non-representative set of vehicles or drivers, the model may overfit to specific vehicle dynamics or driving habits, failing to generalize across the entire customer base.
  • Mitigation Focus: Employing robust cross-validation techniques, extensive regularization during training, and ensuring the training/validation/test splits are geographically, temporally, and fleet-wise diverse.

Data Bias Handling

The Event Reviewer, which uses real-world driving data for classification, is susceptible to various forms of bias. A robust data bias handling strategy focuses on prevention, detection, and mitigation to ensure the model's decisions (True Positive vs. False Positive) are fair and accurate across all operational contexts.

• Source of Bias: Geographical and Environmental Imbalance
  • Challenge: The model may be biased towards the road and weather conditions, lighting, and traffic patterns predominant in the regions where the majority of the training data was collected, leading to uneven performance in underrepresented areas (e.g., poor generalization in snow, fog, or highly specific urban/rural road types).
  • Mitigation: Implement rigorous data collection protocols to ensure a geographically, temporally, and environmentally diverse dataset. Actively sample and prioritize labeling of data from underperforming regions or edge-case environmental conditions.
• Source of Bias: Vehicle/Driver/Fleet Characteristic Skew
  • Challenge: Over-representation of a specific subset of vehicles (e.g., heavy-duty trucks) or driver behaviors could cause the model to perform poorly on other subsets (e.g., light commercial vans or different driving cultures), potentially leading to an unfair or inaccurate safety score calculation for those groups.
  • Mitigation: Monitor the distribution of key fleet characteristics (vehicle type, camera hardware, average mileage) in the training and evaluation sets. Apply re-weighting or stratification techniques during training to ensure the model generalizes across diverse fleet types and driver cohorts.
• Source of Bias: Human Labeling Subjectivity
  • Challenge: The ground truth label (True Positive/False Positive) is derived from human review, where different annotators might have slightly varying interpretations of ambiguous event data, introducing inconsistency or systemic bias in the labels.
  • Mitigation: Establish a clear, unambiguous, and documented labeling rubric. Implement a multi-stage review process (e.g., consensus labeling, expert review) and calculate Inter-Annotator Agreement (IAA) to identify and correct areas of label subjectivity, thereby ensuring the highest quality ground truth.
• Monitoring:
  • Continuous Monitoring: Post-deployment, performance must be monitored and compared across different geographical segments and vehicle/fleet cohorts to detect any emerging disparities in False Positive or False Negative rates, triggering targeted data collection and model retraining.

Model Assumptions and Constraints

Assumptions:

• The AI dashcam hardware and sensors are functioning correctly, providing synchronized, non-corrupted, and calibrated video and auxiliary sensor data for every event trigger.
• The context captured within the short video clip and auxiliary sensor data window is sufficient to determine the event's ground truth (True Positive or False Positive).
• The model's classification task—determining the binary ground truth (True Positive/False Positive)—is stable and remains consistent with the human labeling rubric over the model's deployment lifespan.
• The auxiliary sensor data (speed, accelerometer, GPS) and the video stream are accurately temporally synchronized to enable effective multimodal feature integration.

Constraints:

• Event Latency: There is a real-world time constraint on the end-to-end event review process, as verified events need to be routed to the customer platform promptly for timely safety scoring and review by Fleet Managers.
• Hardware-Defined Input: The model is constrained by the fixed resolution, frame rate, and sensor sampling rate of the AI dashcam hardware. Changes to this hardware would require model re-evaluation or retraining.
• Generalization Boundary: The model's reliability is constrained by its ability to generalize to rare, unforeseen event types, extreme weather, or novel environmental conditions not adequately represented in the training data.
• Computational Budget: The model's complexity (size, number of parameters) and inference time are constrained by the allocated cloud computing resources for the high-volume, continuous event review pipeline.

The Event Reviewer is a binary classification model in a safety-critical and class-imbalanced domain. Its evaluation metrics (KPIs) are selected to ensure high data quality and maximize the identification of genuine dangerous events (True Positives) while effectively filtering out false triggers (False Positives).

Primary Evaluation Metrics (KPIs):

• Precision (on True Positive class): This is a critical customer-facing metric. It measures the proportion of events flagged as "True Positive" by the model that are actually genuine dangerous events. Goal: Maximize Precision to ensure fleet managers and safety officers only see high-quality, actionable events.
• Recall/Sensitivity (on True Positive class): This measures the proportion of all actual dangerous events that the model correctly identified as "True Positive." Goal: Maintain High Recall to ensure the model does not miss genuine safety-critical events (minimizing the False Negative rate).
• F1-Score (on True Positive class): The harmonic mean of Precision and Recall, providing a single score that balances the trade-off between the two. This is a more robust metric than pure accuracy in imbalanced datasets.
• Area Under the Receiver Operating Characteristic Curve (AUC-ROC): Measures the model's ability to distinguish between the two classes across all possible classification thresholds.

Secondary/Monitoring Metrics:

• Overall Accuracy: The proportion of total correct classifications. While useful, it is a poor primary metric due to the class imbalance (a model that always predicts False Positive would have high accuracy).
• False Positive Rate (FPR): The proportion of actual False Positives that the model incorrectly classified as True Positives. This is directly related to the noise seen by the customer and must be kept low.
• False Negative Rate (FNR): The proportion of actual True Positives that the model incorrectly classified as False Positives. This represents missed safety events and is a critical safety metric to minimize.
• Inter-Annotator Agreement (IAA): Monitored on the ground truth data set to ensure the consistency and quality of the human-labeled data used for training.