AI-based Pedestrian Detection and Avoidance at Night using IR Camera, Radar, and Video Camera

On average, a pedestrian is killed every 88 minutes in traffic crashes in the United States. That is more than 16 people a day, almost 115 people a week. The traffic fatalities for the first 9 months of 2020 shows that an estimated 28,190 people died in motor vehicle traffic crashes, which is a 4.5% increase compared to 2019. In 2019, vehicle accidents in the United States killed more than 6,500 pedestrians, the highest annual total ever recorded, and sent more than 100,000 to hospitals with injuries. Additionally, 75% percent of pedestrian fatalities occurred in the dark as compared to daylight (21%), dusk (2%), and dawn (2%). A recent study published by U.S. News & World Report found that roughly 430 Californian pedestrians were tragically killed in the first six months of 2018. Fresno isn’t immune from California’s pedestrian safety woes. According to a 2018 police report, 64% of fatal crashes in Fresno involved both a pedestrian and vehicle. A research study conducted by the Volpe National Transportation Systems Center suggests that automatic emergency braking systems with pedestrian detection functionality could reduce up to 5,000 annual vehicle/pedestrian crashes and 810 fatal vehicle/pedestrian crashes. Pedestrian detection systems with automatic braking functionality have the potential to prevent or reduce the severity of collisions resulting in property damage, personal injury, and/or death.

Principal Investigator: 

Hovannes Kulhandjian

PI Contact Information:

California State University, Fresno


January 2021 to December 2021

Impacts/Benefits of Implementation: 

One possible solution is to use a video camera, a radar system, or a LIDAR system in a vehicle. More recently, the advancement of thermal IR cameras has shown a potential possible solution. The research on pedestrian detection and avoidance is still in its infancy. Several methods have been explored to detect a pedestrian and avoid an accident. To the best of our knowledge, no prior research work has explored or experimented with the idea of using Data Fusion from multiple sensors (i.e., a thermal infrared camera, radar sensor, and a visible camera) combined with advanced Machine Learning (ML) for pedestrian detection and avoiding mechanize. Therefore, we believe that this research exploration could lead to new Artificial Intelligent-based application tools for drivers that could potentially save lives. We will be exploring state-of-the-art ML techniques combined with data fusion to achieve this objective. The goal of this research work is to maximize the detection capabilities of pedestrians, especially at night, by effectively data fusing the information gathered from a thermal camera, a radar sensor, and a video camera along with the use of advanced machine learning algorithms to detect and avoid pedestrian collision in real-time. Using this multi-dimensional valuable data, it could make intelligent decisions during different conditions of the road, be it during the day or at night. The proposed system could potentially be embedded into a smart vehicle system that provides real-time pedestrian detection and alerting mechanism by vibrating the driver’s wheel and display a message on a monitor/dashboard to warn the driver to avoid colliding into the pedestrian. The proposed system can be used both during the day and at night using the combination of a thermal infrared camera, a radar system, and a video camera. It could also be installed in an autonomous vehicle.

Project Number: