This section outlines the key aspects of the data used to develop the model. First, we describe the data and its features. Then, the pipeline and its specifics are explained.
Description on Data and Features
This section details the data used in the Driver Monitoring System (DMS), including the types of information captured and the features derived for driver behavior analysis.
Data Types and Sources:
- Dashcam Video Feed: Real-time video captured from the in-cabin dashcam, mounted on the vehicle windshield, facing the driver. This is the primary data source for visual analysis of driver state.
- Audio Data (Optional/Future Enhancement): Sound captured within the cabin (e.g., driver speech patterns, yawning sounds) to potentially supplement visual cues for drowsiness. This is currently not a primary input but may be considered for future versions.
- Vehicle Telematics Data (Integration): Data from the vehicle's onboard systems (e.g., speed, acceleration, braking, steering wheel angle) could be integrated to provide contextual information about driving conditions and correlate with driver behavior. This data is not directly captured by the dashcam but is crucial for comprehensive driver evaluation.
Key Features Extracted from Video Feed:
The DMS extracts a variety of visual features to infer driver state and behavior. These include:
- Face Detection and Tracking:
- Features: Presence and location of the driver's and passenger's faces.
- Purpose: To confirm driver/passenger presence and ensure the driver is facing the road.
- Eye Gaze and Eye Closure Estimation:
- Features: Eye openness (e.g., PERCLOS - Percentage of Eye Closure over time), eye blinking frequency, gaze direction.
- Purpose: Primary indicators for drowsiness, inattention (eyes off road), and distraction.
- Head Pose Estimation:
- Features: Real-time orientation of the driver's head in 3D space (e.g., looking down, looking away, nodding).
- Purpose: To detect distraction (head turned away from road), drowsiness (head nodding), and inattention.
- Facial Landmark Detection:
- Features: Key points on the face (e.g., corners of eyes, mouth, nose).
- Purpose: Supports robust face and eye tracking, and can be used for detecting micro-expressions or specific signs of fatigue (e.g., yawning).
- Mouth and Yawning Detection:
- Features: Mouth openness, characteristic shape of a yawn.
- Purpose: Direct indicator of drowsiness.
- Hand and Object Detection (within cabin):
- Features: Detection of hands near the face, hands holding objects (e.g., mobile phone, food/drink).
- Purpose: To identify specific types of manual distraction (e.g., phone use while driving, eating/drinking).
- Eyewear Detection:
- Features: Presence or absence of eyeglasses or sunglasses.
- Purpose: To adapt eye tracking algorithms for various lighting conditions and to account for potential occlusions.
- Cabin Object Detection:
- Features: Identification of common objects within the vehicle cabin that could contribute to distraction or indicate specific activities.
- Purpose: Contextual understanding of driver interactions within the cabin.
- Seatbelt Detection:
- Features: Presence and correct use of seatbelt.
- Purpose: To monitor compliance with safety regulations and provide alerts if not buckled.
- Eye Rubbing Detection:
- Features: Detection of hand-to-eye contact and rubbing motion near the eyes.
- Purpose: Additional drowsiness/fatigue indicator, complementing PERCLOS and blink-rate signals.
- Hood/Head Covering Detection (Hood-Up):
- Features: Binary classification of a hood pulled up over the head. The negative class contains normal heads, caps and beanies, long hair, head scarves and similar head coverings that are not a hood (special_headscarf), and ambiguous frames, which are deliberately labelled negative so that they never trigger an alert.
- Purpose: To flag facial occlusion that can degrade face, eye, and head-pose model accuracy, and to detect a specific non-compliant/evasive driver behavior.
Derived Features and Behavioral Patterns:
From the raw visual features, the system derives higher-level behavioral patterns and risk indicators:
- Distraction Events: Duration and frequency of eye-off-road, head-turned-away, or hand-held device use.
- Drowsiness Indicators: Sustained periods of eye closure (PERCLOS), frequent or prolonged blinking, yawning detection, eye-rubbing frequency, and micro-sleep events, consolidated into a Karolinska Sleepiness Scale (KSS) score via a temporal SVR model.
- Inattention: Lack of consistent engagement with the driving task, characterized by a combination of the above.
- Driving Pattern Summaries: Aggregated data over time to assess overall driver attentiveness, distraction levels, and fatigue risk.
Initial DMS inference (e.g., face, eye, and head-pose analysis) is conducted on the edge device (dashcam) to minimize latency for real-time alerts. Whether raw video, event clips, or only derived event metadata is transmitted to the cloud is governed by the customer's (Data Controller's) recording and privacy settings in MyGeotab; see Data Privacy for details.
Data Pipeline and Preprocessing
Data Ingestion and Source
The Driver Monitoring System (DMS) primarily ingests data directly from an in-vehicle dashcam and, where available, integrates with vehicle telematics. The design prioritizes edge processing to ensure real-time performance and data privacy.
- Primary Data Source: In-Cabin Dashcam
- Device: A dedicated dashcam unit installed on the vehicle's windshield, positioned to have a clear view of the driver's face and upper body within the cabin.
- Input: High-resolution video stream (e.g., 1080p at 30 FPS) from the dashcam's in-cabin camera. This stream is continuous while the vehicle is in operation and the DMS is active.
- Direct Access: The DMS software runs directly on the dashcam's embedded processor, allowing direct access to the raw video frames as they are captured by the camera sensor.
- Infrared (IR) Capabilities: The dashcam is equipped with IR illumination and an IR-sensitive camera to ensure robust performance in all lighting conditions, including night driving, low light, and when the driver is wearing sunglasses.
- Secondary Data Source (Integration): Vehicle Telematics
- Integration Method: While not directly ingested by the dashcam's camera sensor, vehicle telematics data can be integrated with the DMS system through a separate connection (e.g., GO Device, CAN bus, OBD-II port, or via a connected telematics gateway).
- Input: Real-time vehicle operational data, including:
- Vehicle speed
- Braking events (e.g., harsh braking)
- Acceleration events (e.g., harsh acceleration)
- Steering wheel angle
- GPS location (for contextual analysis of driving environment)
- Purpose of Integration: To provide contextual information that enriches the visual analysis from the dashcam. For example, knowing vehicle speed or braking intensity can help better interpret driver behaviors (e.g., sudden gaze shift during harsh braking). This data is primarily used for post-trip analysis and overall driver scoring rather than real-time alerts from the DMS itself.
- Data Ingestion Mechanism:
- Edge Processing: All raw video data is ingested and processed directly on the dashcam device itself (at the "edge"). This local processing is critical for:
- Low Latency: Enabling real-time detection and immediate alerts to the driver.
- Data Privacy: On-device processing reduces the need to transmit raw video, but does not eliminate it. Whether raw video, event clips, or only derived features/metadata leave the device is determined by the customer's (Data Controller's) Continuous or Event Recording settings.
- Streaming Data: The video feed is continuously streamed from the camera sensor to the DMS processing modules within the dashcam's firmware.
- Controller-Configured Data Transmission: Depending on Controller settings, transmission to Geotab's secure cloud platform ranges from event metadata and derived labels (which the Controller can link to a vehicle and driver and which are therefore handled as personal data) up to full event-triggered or continuous raw video upload. Raw video, where uploaded, is stored in access-controlled, encrypted, audit-logged cloud storage. Training data is obtained by downloading selected customer events through the Geotab annotation platform to the Computer Vision team's on-premises training server; at download the driver's display name and top-level driver id are removed from the event metadata and, for cabin-camera events, GPS rows are removed from the sensor files, while a pseudonymous numeric driver id is retained to allow driver-independent evaluation splits (being introduced, Oct 2026).
Data Preprocessing for Model Training
The robust performance of the DMS model relies heavily on meticulously prepared training data. This offline preprocessing pipeline transforms raw video and sensor data into high-quality, labeled datasets suitable for supervised machine learning.
- Raw Video Collection:
- Diversity is Key: Collect vast amounts of video footage from diverse real-world driving scenarios. This includes:
- Driver Demographics: Training data reflects the customer fleets it is drawn from; demographic composition is not measured because demographic attributes are not collected. See Data Bias Handling.
- Lighting Conditions: Varying light levels (bright sunlight, overcast, twilight, night driving, tunnels), shadows, glare, and reflections. The use of IR cameras ensures data capture in challenging low-light conditions.
- Driving Scenarios: Urban, rural, highway, varying traffic densities, and different vehicle speeds.
- Camera Placement Variations: Account for slight differences in dashcam installation angles and positions.
- Driver Behaviors: Specifically capture instances of:
- Attentive Driving: Normal, focused driving.
- Distraction: Phone use (holding, looking at), eating, drinking, interacting with infotainment, looking at passengers, turning to talk.
- Drowsiness: Eye closure (micro-sleeps), frequent blinking, prolonged blinks, yawning, head nodding, head droop.
- Inattention: Gaze off-road for extended periods, eyes not tracking the road.
- Data Annotation and Labeling:
- Human-in-the-Loop Annotation: Highly skilled human annotators are employed to meticulously label features and behaviors within the collected video frames. This is a critical and labor-intensive step.
- Frame-level Labels: For each relevant frame or sequence of frames, the following are precisely marked:
- Bounding Boxes: Accurate rectangular regions identifying the driver's face, eyes, mouth, hands, and relevant objects (e.g., mobile phones, food).
- Facial Landmarks: Key points (e.g., 68-point or more extensive landmarks) on the face and eyes, crucial for head pose and eye status estimation.
- Eye State: Categorical labels for eye openness (e.g., "open," "half-closed," "closed") and gaze direction (e.g., "on-road," "left mirror," "center console," "phone").
- Head Pose: Explicit 3D pose (pitch, yaw, roll) annotations for the driver's head.
- Mouth State: "Open" or "closed," with specific labels for "yawning."
- Hand State: "On steering wheel," "holding phone," "eating," "other object."
- Seatbelt Status (driver seat, 4 classes): fastened_safely, fastened_unsafely, unclear, unfastened.
- Environmental Context: Rough labels for lighting conditions (e.g., "day_bright," "day_low_light," "night_IR").
- Temporal Behavior Labels: Beyond static frame labels, annotate the duration and onset/offset of continuous driver behaviors:
- "Distracted_Phone_Use" (start/end timestamp)
- "Drowsy_MicroSleep" (start/end timestamp)
- "Inattentive_GazeOffRoad" (start/end timestamp)
- Quality Control: Labels are produced by trained annotators on the annotation platform and by the GVP Computer Vision Team. A data-quality check (completeness, validity, uniqueness including train/validation leakage, image integrity) is run before a dataset is used for training and its report is kept with the dataset. Multi-annotator agreement is not currently measured.
- Data Cleaning and Validation:
- Anomaly Detection: Identify and remove corrupted frames, video artifacts, or severely mislabeled examples.
- Consistency Checks: Verify logical consistency across labels (e.g., if "eyes closed" is labeled, "gaze on-road" should not be the primary label).
- Outlier Removal: Address any extreme outliers in feature values that might stem from annotation errors or sensor glitches.
- Data Augmentation:
- Synthetically Increase Data Volume: Crucial for improving model generalization and robustness, especially for rare events.
- Geometric Transformations: Apply random rotations, translations, scaling, cropping, and horizontal flipping (for symmetrical features like face detection) to simulate variations in camera perspective and driver position.
- Photometric Transformations: Adjust brightness, contrast, saturation, hue, and add various types of noise (e.g., Gaussian noise, salt-and-pepper noise) to simulate diverse lighting and environmental conditions.
- Occlusion Simulation: Artificially introduce realistic occlusions (e.g., parts of the steering wheel, hands, sun visors, sunglass reflections) over the face and eyes to train the model to be robust to partial visibility.
- Temporal Augmentation: Vary the speed or duration of behavioral sequences.
- Feature Engineering (as applicable):
- For some model components or early stages of development, hand-crafted features might be engineered from the raw data. However, for deep learning models, the focus is on providing raw inputs and letting the model learn hierarchical features.
- Dataset Splitting:
- Training Set: The largest portion of the data, used to train the model parameters.
- Validation Set: Used during training to tune hyperparameters and monitor for overfitting. It provides an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters.
- Test Set: A completely unseen dataset, used only once at the very end of the development cycle to provide a final, unbiased evaluation of the model's performance on new data.
- Strategic Splitting: Validation sets are dated, separately collected folders; splits are not currently driver-independent. Driver-independent splitting is planned on the retained pseudonymous driver id.
- Data Storage and Management:
- Preprocessed and labeled training data is stored on the Computer Vision team's on-premises training server (access-controlled NVMe storage); dataset versions are being registered in ClearML together with their data-quality report.
- Implement strict access controls and encryption to protect sensitive data.
- Maintain clear metadata for each dataset (e.g., collection date, camera type, annotation guidelines version) to ensure reproducibility and traceability.
- Training Data Residency: Training data is currently US-sourced and held in the US training environment; no EU training dataset exists at this time.
- No PII in Model Outputs: The output of the training process consists only of optimized model weights, not personal data. Individual information is not embedded in, identifiable within, or recoverable from the resulting models.
Model Training and Storage
This section details the processes involved in training the Driver Monitoring System (DMS) models and how these models, along with associated data, are securely stored and managed throughout their lifecycle.
- All the underlying models are trained independently using their respective preprocessed datasets.
- Training is conducted on high-performance computing infrastructure (e.g., GPUs).
- Trained model weights and configurations are version-controlled in an internal repository (Git LFS) with an md5 recorded for every release package; detector checkpoints are additionally tracked in ClearML. Each production release is authorised through a release deployment authorization (named approvers including the Head of Computer Vision) after a bench-test comparison against the previous release.
Post-processing for Forecast Output
The raw outputs from the DMS inference engine undergo several post-processing steps to generate actionable insights and user-friendly reports. This ensures that the "forecast" of driver behavior is accurate, interpretable, and relevant for real-time alerts and long-term evaluation.
- Thresholding and Event Filtering:
- Raw model outputs (e.g., probabilities for distraction, drowsiness scores) are compared against predefined, empirically tuned thresholds.
- Minor or transient fluctuations that do not indicate a significant safety risk are filtered out to prevent false positives and alert fatigue.
- Example: A brief glance away (less than 1 second) might be ignored, while a sustained gaze off-road for 3+ seconds triggers an alert.
- Temporal Smoothing and State Consolidation:
- Outputs over short time windows are stabilised by multi-window statistics, confidence tracking and state-flip-rate detection, a temporal-inference layer (RandomForest / CatBoost per behaviour) and a rule evaluator with per-behaviour thresholds; a per-driver behaviour calibration baseline is used for the microsleep model.
- Consecutive detections of the same event (e.g., multiple frames showing "drowsy") are consolidated into a single, longer event with a clear start and end time.
- Example: Instead of multiple "drowsy" alerts for 5 seconds, it's consolidated into one 5-second "drowsy episode."
- Severity Scoring:
- Detected events are assigned a severity score based on duration, frequency, and contextual factors (e.g., vehicle speed, time of day).
- Example: A 10-second distraction event at highway speeds is more severe than a 3-second distraction at low speed.
- Contextual Integration:
- Outputs from various DMS modules (e.g., eye gaze, head pose, hand detection) are combined with any available telematics data (speed, braking) to provide a richer understanding of the driving context.
- Example: A driver's head turning might be benign if the vehicle is stopped, but critical if at high speed.
- Alert Prioritization and Management:
- For real-time alerts, a logic system prioritizes simultaneous or competing alerts to avoid overwhelming the driver (e.g., a critical drowsiness alert might supersede a minor distraction alert).
- Voice alerts are triggered with clear, concise messages.
- Data Aggregation for Reporting:
- For long-term evaluation, raw event data is aggregated into meaningful summaries:
- Trip-level Reports: Total distraction time, number of drowsiness events, risky driving scores per trip.
- Driver-level Profiles: Historical trends, average scores, and identified patterns over days/weeks/months.
- Fleet-level Analytics: High-level overview of safety trends, identification of high-risk drivers or routes.
- Data minimisation (cloud transmission):
- Raw cabin frames are processed on the device and are not transmitted except as part of an uploaded event clip. Derived metadata (behavioural labels, scores, event counts) is pseudonymous at the platform level but can be linked to a vehicle and driver by the Controller; it is therefore handled as personal data under the Controller/Processor agreement and the platform's retention and access controls.
- Visualization and User Interface Generation:
- The processed and aggregated data is then used to populate dashboards, reports, and mobile applications, providing fleet managers and drivers with clear visualizations and actionable insights into driving behavior and safety performance.
Data Privacy
The Driver Monitoring System (DMS) runs on the Geotab Video Platform and inherits that platform's privacy-by-design architecture and Data Controller / Data Processor governance model.
- Data Controller / Data Processor Model: The customer (fleet operator) is the Data Controller — they own the video data and determine the purpose and means of processing, including which events trigger a cloud upload, how long data is retained, and how driver consent is obtained. Geotab acts as the Data Processor, providing the secure technical infrastructure and processing data solely on the Controller's instructions.
- Controller-Configured Recording & Minimized Transmission: Video is captured and initially processed on the dashcam. Depending on Controller-defined settings, video is either recorded locally with cloud upload occurring on request or triggered by a predefined safety exception ("Continuous Recording"), or automatically uploaded when a specific DMS event occurs, such as distraction or drowsiness ("Event Recording"). Raw video is not continuously streamed to the cloud; transmission is limited to Controller-configured events or requests.
- AI-Driven Object Blurring: To support Controller privacy obligations, the platform can apply AI-based blurring of personally identifiable visual elements, configurable in MyGeotab: license plates and pedestrian faces (road-facing view), and driver and passenger faces (cabin view).
- Derived Metadata: Behavioral features and event metadata generated by the DMS models (e.g., "distraction detected," drowsiness score, event counts) do not contain the driver's name or image, but the Controller can link them to a vehicle and driver; they are handled as personal data.
- Encryption: Video is encrypted using AES-256 at rest, both on the camera's local storage and in Geotab's cloud storage, and using TLS 1.3 in transit, with mutual authentication ensuring only verified cameras can communicate with the platform.
- Data Residency: Standard video and event data are stored in data centers corresponding to the customer's region (e.g., US data in the US, EU data in the EU). Data selected for model training is currently US-sourced and held in the US training environment. On-device state (per-driver calibration and the 48-hour event-count history) stays on the camera, rolls over continuously and is uploaded only as the per-event counts described under Intended Use. Privacy assessments: a DPIA covers the in-cabin system; the DMS AI risk assessment is in progress.
- Consent Management: Because the customer is the Data Controller, obtaining explicit and informed consent from drivers — for example, for DMS monitoring or for the capture of event-triggered video clips — is the customer's responsibility. Geotab provides the technical controls (recording settings, blurring, retention, access management) that support the Controller in meeting this obligation, but Geotab does not itself collect driver consent.
- Secure Storage & Access Control: Raw video is held in access-controlled, audit-logged storage, viewable only by the Controller's authorized MyGeotab users (via role-based security clearances such as View Live Video / View Recorded Video) and a small, restricted set of Geotab personnel. Company Groups further silo access so users only see footage for drivers and vehicles within their assigned groups.
- Retention & Deletion: When a MyGeotab database or driver is removed, an automated process deletes the associated video and derived data in accordance with Geotab's applicable data-retention and deletion policies.
- Purpose Limitation: Data collected by the DMS is used solely for its intended purpose — driver safety alerting and driving-pattern evaluation as configured by the Controller — and not for unrelated purposes without the Controller obtaining separate, purpose-specific consent (e.g., a future insurance use case would require new consent, distinct from the original safety-monitoring consent).
- Compliance: The system adheres to relevant data protection regulations and privacy laws (e.g., GDPR, CCPA) through the technical and organizational measures above.