Test Results: AEB Fused with Thermal Can Save Lives

Download the full whitepaper

At the end of 2019, FLIR and VSI joined forces to advance the use of thermal cameras for the development of a more reliable automatic emergency braking (AEB) system in all conditions. 

In July 2020, we returned to the track at the American Center for Mobility (ACM) to test a fused AEB system that included a thermal camera, radar, visible camera, and FLIR CNN. The tests were conducted alongside four commercially available vehicles in challenging conditions.

The test results show that the AEB system with the added thermal camera performed significantly better than existing commercial AEB systems in the supplemental gap testing. Read the Executive Summary of the test results below or download the full whitepaper.

This video is unavailable due to your cookie settings.

---

Automatic Emergency Braking Fused with Thermal Can Save Lives

Existing AEB systems are powered by radar and visible cameras and none feature a thermal imaging camera. In our tests, we showed that Teledyne FLIR thermal cameras and the Research Vehicle consistently stopped safely in challenging conditions.

Executive Summary: Improving AEB Effectiveness with Thermal Sensing Technology

Current Conditions

In 2019, vehicle accidents in the United States killed more than 6,000 pedestrians, the highest annual total ever recorded, and sent more than 100,000 to hospitals with injuries. 75% of the reported fatalities occurred at night. (1) In an ongoing effort to make roads safer, twenty automakers committed to the U.S. Congress that all newly manufactured cars and trucks will be equipped with AEB systems starting in 2022. However, today’s Euro New Car Assessment Programme (NCAP) testing procedures do not include testing for common driving conditions such as driving in total darkness or with glare. The AAA exposed potential weaknesses in current AEB systems in day and night conditions in their October 2019 paper, Automatic Emergency Braking with Pedestrian Detection. (2)

To detect pedestrians, current AEB systems rely on either visible light cameras, radar or both. None currently use thermal cameras, which can detect a pedestrian much further than typical headlights can illuminate for a visible camera.

TESTING THERMAL TECHNOLOGY IN AEB SYSTEMS

This paper tests the theory that adding thermal technology to today’s radar and visible light technology can reduce pedestrian collisions in common driving conditions. A FLIR sponsored car using a fused AEB system combining thermal, radar, and visible light sensing with a convolutional neural network (CNN) tested against four commercially available cars with state of the art AEB systems. Five test cases were developed based on Euro NCAP testing protocols. The tests included scenarios not currently in standard AEB positive detection testing. These are: daytime when pedestrians’ clothes blend in with the background, daytime pedestrian in oversized clothing, nighttime driving, driving into sun glare, and nighttime child and adult pedestrian emerging from behind a parked car.

The results are compelling:
• The fused thermal AEB system was successful in 25 of 25 tests at effectively preventing pedestrian injury with only two instances where the vehicle contacted, but did not knock down the SPT (soft pedestrian target)
• The four commercially available AEB systems had positive performance in daytime tests (42 passed out of 50 tests). These systems did not perform well in the nighttime tests, striking the SPT in all but two test cases.
• There is an opportunity for automotive manufacturers, Tier 1 suppliers, government regulators, and automotive testing agencies to make AEB more reliable by expanding AEB compliance testing to include viable sensing technologies such as thermal sensing.

Introduction

For several years, FLIR Systems, Inc. (FLIR) has been working with auto manufacturers and Tier 1 suppliers to assist in the development of cutting-edge ADAS and AV Systems with thermal Far Infrared (FIR) in the sensor suite. FLIR engaged VSI Labs (VSI) to develop and test the world’s first fused AEB sensor suite that employed a thermal longwave infrared (LWIR) camera, a radar, a visible camera, and a convolutional neural network (CNN).

VSI incorporated this system into a 2018 Ford Fusion and tested it along with four 2019 vehicles deployed with then state of the art AEB systems. All tests were conducted at the American Center for Mobility (ACM) in Ypsilanti, Michigan. The tests were based on Euro New Car Assessment Programme (NCAP) positive detection tests with additional gap tests developed by VSI to represent common driving conditions such as darkness and sun glare. These gap tests represent conditions not currently tested by NCAP or other testing agencies. VSI designed the tests to include driving toward a soft pedestrian target (SPT) heated to mimic a human at 25 mph (40 kph).The FLIR premise was that current commercially available vehicle AEB systems would struggle to identify and react to a pedestrian target in dark and sun glare conditions and with completely bright or dark oversized clothing.

This paper describes VSI’s testing methodology, procedures, and results.

SENSING TECHNOLOGIES FOR AEB

FLIR believes there is no single sensor that can enable a reliable Automatic Emergency Braking (AEB) system. It is only the fusing of a combination of sensors that will enable an AEB system to perform accurately in common driving conditions. As the results of the tests shown in this paper demonstrate, a combination of thermal, radar, and visible thermal sensors with pedestrian detection has the potential to do so.

Methodology 

The most common pedestrian collisions occur when a pedestrian crosses a roadway perpendicular to a vehicle. Most Euro NCAP pedestrian AEB test cases emulate this type of scenario3. VSI based their test protocols on this perpendicular pedestrian crossing scenario, with the addition of five gap tests that include testing in darkness, sun glare, and alternate clothing a pedestrian may wear. The NCAP standards were followed in the daytime testing with dry road conditions, the temperature between 5° to 50° Celsius (41° to 122° Fahrenheit), daylight, and clear visibility. At night key NCAP procedures were followed to emulate the car perpendicular position with the addition of the nighttime, gap test features. To reduce the risk of damage to the vehicle or SPT with repeated collisions, each test case was repeated five times or until the vehicle under test (VUT) collided with the SPT a maximum of two times.

At the ACM, the VSI Ford Fusion with the fused thermal, radar and visible sensors was subjected to the Euro NCAP tests that commercially available cars pass during development and as part of a new car assessment program. The tests were administered by the ACM track staff. The VSI Ford Fusion passed all tests, the complete results can be found in Appendix II of this paper.

COMMERCIALLY AVAILABLE VEHICLES WITH AEB

VSI selected four commercially available vehicles (4) with top AEB systems, to test alongside the VSI Ford Fusion test car that included the thermally fused AEB system. The vehicle models were:

2019 Tesla Model 3 with Autopilot™ 3.0

The 2019 Tesla Model 3 is an electric sedan with Autopilot 3.0. The Tesla’s AEB system is based on three forward-facing cameras behind the windshield in the rearview mirror cutout and a 160-meter forward facing radar. Learn more at Tesla Autopilot. 

2019 Toyota Corolla with Toyota Safety Sense™ 2.0

The 2019 Toyota Corolla is a four-door sedan with the Toyota Safety Sense 2.0 safety system that includes an integrated forward-facing visible camera and grille-mounted radar system designed to mitigate a collision with a preceding car or pedestrian. Safety Sense 2.0 is the latest generation ADAS package and claims improved detection in low-light conditions. (5) Learn more at Toyota Safety Sense 2.0. 

2019 Subaru Forester with Subaru EyeSight®

The 2019 Subaru Forest is a compact SUV equipped with Subaru EyeSight driver assist technology. The system is based on two forward-facing cameras mounted behind the windshield on either side of the rearview mirror. Learn more at Subaru EyeSight. 

2019 BMW X7 with Mobileye®

The BMW X7 is a full-size SUV with Mobileye’s TriCam4 module. The AEB system with pedestrian and cyclist detection is based on three forward-facing cameras with 28º, 52°, and 150° field of views. Learn more at BMW Driver Assistance. 

VEHICLES TESTED AT ACM

To read the rest of the paper, click the download link below: 

Download the full whitepaper

(1) https://www.ghsa.org/resources/news-releases/pedestrians20
(2) https://www.aaa.com/AAA/common/aar/files/Research-Report-Pedestrian-Detection.pdf
(3) M. Yanagisawa, E. Swanson, P. Azeredo, and W. Najm, "Estimation of potential safety benefits for pedestrian crash avoidance systems," National Highway Traffic Safety Administration, 2017.
(4) All product and company names are trademarks™ or registered® trademarks of their respective holders. Use of any product or company trademark does not imply any affiliation with FLIR or VSI, or any endorsement by the mark owners.
(5) https://www.toyota.com/content/ebrochure/CFA_TSS_2.pdf