Detecting Driver Drowsiness with Multi-Sensor Data Fusion Combined with Machine Learning

According to the National Highway Traffic Safety Administration (NHTSA), 91,000 police-reported crashes involved drowsy drivers in 2017 alone. These crashes led to an estimated 50,000 people injured and claimed 795 lives. However, these numbers are underestimated, and up to 6,000 fatal crashes each year may be caused by drowsy drivers. According to a Centers for Disease Control and Prevention (CDC) study, 1 in 25 drivers surveyed reported that they had fallen asleep while driving in the past 30 days. Studies have shown that not having enough sleep, hence being drowsy, can impair your ability to drive the same way as drinking too much alcohol. “Drowsy drivers put themselves and others at risk through a slower reaction time and the inability to pay attention,” said California Highway Patrol (CHP) Commissioner Warren Stanley at a Press Release. He went on and added, “A sleepy driver can be just as impaired or dangerous as one under the influence of alcohol or drugs”. In addition to that in the same Press Release California Department of Transportation (Caltrans) Caltrans Director Toks Omishakin said, "In a state the size of California, long drives between cities are common. Without enough rest, all of us may feel drowsy behind the wheel”. According to the California Office of Traffic Safety, signs of driver fatigue are mainly contributed to yawning, blinking frequently as well as daydreaming.

One possible solution is to enable the vehicle to detect drowsiness or discrepancies in the driver’s behavior and alert the user when it occurs. The research on detecting drowsy drivers and alerting them is still in its infancy. Several methods have been explored to detect driver drowsiness with the intention to alert them.  To the best of our knowledge, no prior research work has explored or experimented with the idea of using Data Fusion (DF) from multiple sensors (i.e., video camera with night vision capabilities and micro-Doppler radar) combined with Machine Learning (ML) for drowsy driver detection and alerting mechanize. Therefore, we believe that this research exploration could lead to new Artificial Intelligent-based application tools for drivers that could potentially save lives. We will be exploring state-of-the-art ML techniques combined with DF to achieve this objective. The goal for this research work is to maximize the detection accuracy of drowsy driving and alert them by effectively data fusing the information gathered from a video camera with night vision capabilities and micro-Doppler radar along with the use a trained Deep Convolutional Neural Network (DCNN) system to classify and identify drowsy driving features in real-time. Using this multi-dimensional valuable data, it could make intelligent inferences about the driver's behavior and alert drowsy driver from falling asleep and having or causing a fatal accident. The proposed system could potentially be embedded into a smart car system that provides real-time drowsy driving alerting mechanism by vibrating the driver’s wheel and a display a message on a monitor/dashboard (e.g., “Stay Awake to Stay Alive”) to warn the driver of falling asleep and recommend to pull over to have some rest. The proposed system can be used both during the day and at night using the combination of a video camera with night vision capabilities and the micro-Doppler radar sensor.

Principal Investigator: 

Hovannes Kulhandjian

PI Contact Information:
California State University, Fresno

Project Number: