A research team led by Fuyuki Ishikawa at the National Institute of Informatics (NII, Japan) has developed a technique to search automatically for simulation configurations that test various behaviors of automated driving systems. This research was conducted under the ERATO-MMSD project funded by the Japan Science and Technology Agency (JST, Japan). The proposed technique iterates trials on simulations using an optimization method called evolutionary computation so that it discovers simulation configurations that lead to specific features of driving behaviors such as high acceleration, deceleration, and steering operation. The outcome of this research was presented in ICST 2021, a flagship conference on software testing held during April 12-16 2021.
Background
More attention is being focused on automated driving systems (ADS) or advanced driver assistant systems. New car models with Level 3 of autonomous driving are emerging, ones that do not require human drivers to supervise the driving operation under certain conditions. However, the ADS functionality being put into practical use is limited to specific situations such as traffic jams on highways or fixed routes. Increases in safety and reliability are required for use of ADS in environments with enormous situations such as urban areas.
One of the key functions in ADS is path planning, which continuously updates direction and speed by examining the surrounding environment, including other cars and pedestrians. The path-planning functionality needs to handle not only safety but also multiple aspects such as the extent of acceleration/deceleration, steering operation, and lane conformance.
Simulation-based testing is commonly used for ADS. A typical approach is that human testers enumerate scenarios. An example is “the ego-car is going to take a right turn, but a car is approaching from the opposite direction.” However, the ADS behavior can differ in the same right turn scenario, for example, either taking a turn without the need for braking or decelerating and waiting for a long time before taking the turn. It is essential to check different behaviors the ADS can take before utilizing it in society. However, specific behaviors such as long deceleration are unlikely to occur when researchers run many simulations under configurations with different positions of other cars and so on. Moreover, the ADS has more possible specific behaviors, for example, simultaneous occurrences of strong acceleration and high amounts of steering operation. Configuring simulations to cause such specific behaviors intentionally is very difficult.
In this research, the researchers proposed a technique for test generation that automatically searches for simulation configurations leading to specific features of driving behaviors such as high acceleration and deceleration and high amounts of steering operation. They used an optimization technique called evolutionary computation, which repeats simulation trials to adjust configurations so that specified driving behaviors last for a long period of time. In this way, the technique can discover simulation configurations, such as the positions of other cars, leading to the desired features of driving behaviors.
The proposed technique also avoids only generating simulation configurations that only lead to dangerous situations such as collisions. Therefore, it reveals features of driving behaviors not limited to emergency situations. In addition, it can search for and trigger combinations of behaviors such as simultaneous occurrences of high acceleration and high amounts of steering operation.
We applied and evaluated the test generation technique to a program of path planning offered by Mazda. The technique could generate specific behaviors that were rarely caused in random simulations. For example, it generated strong acceleration together with high amounts of steering operation as well as high acceleration following high deceleration in a scenario for a right turn at an intersection. These cases occurred only with very specific timings of other cars entering the intersection. In this way, the researchers showed the technique can intentionally trigger combinations of specific behaviors using simulation configurations that are very difficult for human engineers to design.
Future outlook
This research was conducted in the JST ERATO-MMSD project. In the project, the researchers investigated other techniques for discovering simulation scenarios that lead to crashes, techniques that explain the causes of crashes, and techniques that fix the behaviors to avoid the detected crashes. The research this time was to increase confidence in the system safety by checking various situations, in addition to the techniques for detecting and fixing problematic behaviors. Thus, the researchers established a comprehensive approach for testing of ADS with both tests for detecting problems and tests for checking diverse cases, which have been done for conventional software programs.
Late 2020 featured a competition for test generation tools on advanced driver-assistance systems (ADAS) (in conjunction with the SBST Workshop to be held in May 2021). The ERATO-MMSD project submitted a tool called Frenetic to the competition. Frenetic made significant results in terms of the rates of generated failure cases and their diversity. This exactly came from the aforementioned research experience.
The scientists provided comprehensive testing techniques for ADS. Although they used the program provided by Mazda in the evaluations, the techniques are generic and can be tailored for the specific demands of each automotive company. For example, they can adjust the techniques to the emerging framework called responsibility-sensitive safety proposed by Intel and Mobileye. They will endeavor to make the techniques available by tailoring them for emerging international standards, as well as the demands from each automotive company.
Comment by Fuyuki Ishikawa
“We have conducted active research on the path-planning component through collaboration with Mazda. We have established a holistic set of testing and debugging techniques, including the aforementioned one, by adapting techniques for conventional program code. The key of these techniques is to search for solutions such as desirable tests and desirable fix actions. We will extend and empirically validate the techniques given emerging standards as well as different demands in each ADS application.”