A Scenario-Based Test Selection and Scoring Methodology for Inclusion in a Safety Case Framework for Automated Vehicles

Effectively determining automated driving system (ADS)-equipped vehicle (AV) safety without relying on testing an infeasibly large number of driving scenarios is a challenge with wide recognition in industry and academia. The following paper builds on previous work by the Institute of Automated Mobility (IAM) and Science Foundation Arizona (SFAz), and proposes a test selection and scoring methodology (TSSM) as part of a safety case-based framework being developed by the SFAz to ensure the safety of AVs while addressing the scenario testing challenge. The TSSM is an AV verification and validation (V&V) process that relies, in part, on iterative, partially random generation of AV driving scenarios. These scenarios are generated using an operational design domain (ODD) and behavioral competency portfolio, which expresses the vehicle ODD and behavioral competencies in terms of quantifiable amounts or intensities of discrete components. Once generated, these scenarios are subjected to filters based on their relevance to the AV ODD and behavioral competency portfolio that preserves the robustness of the generated test set; after filtration, scenarios are assigned to a test method and executed. Further, these scenarios may be generated entirely by the TSSM or may be drawn from a preexisting scenario database and subjected to the same filtration process. After the scenarios assembled by the TSSM are executed, the methodology aggregates their driving assessment (DA) scores into a single numerical value. We outline the overall safety case-based framework, the TSSM, including its role in the framework as well as planned future work, and outline two proofs of concept: (1) a demonstration of the ability of the TSSM to pare down the space of scenarios in a scenario database; and (2) a specification form which may be used to solicit a description of the AV ODD and behavioral competency portfolio from the AV developer.