A Scenario-Based Test Selection and Scoring Methodology for Inclusion in a Safety Case Framework for Automated Vehicles

Gavin O'Malley, Jeffrey Wishart, Junfeng Zhao, Brendan Russo

Research output: Contribution to journalConference articlepeer-review


Effectively determining automated driving system (ADS)-equipped vehicle (AV) safety without relying on testing an infeasibly large number of driving scenarios is a challenge with wide recognition in industry and academia. The following paper builds on previous work by the Institute of Automated Mobility (IAM) and Science Foundation Arizona (SFAz), and proposes a test selection and scoring methodology (TSSM) as part of a safety case-based framework being developed by the SFAz to ensure the safety of AVs while addressing the scenario testing challenge. The TSSM is an AV verification and validation (V&V) process that relies, in part, on iterative, partially random generation of AV driving scenarios. These scenarios are generated using an operational design domain (ODD) and behavioral competency portfolio, which expresses the vehicle ODD and behavioral competencies in terms of quantifiable amounts or intensities of discrete components. Once generated, these scenarios are subjected to filters based on their relevance to the AV ODD and behavioral competency portfolio that preserves the robustness of the generated test set; after filtration, scenarios are assigned to a test method and executed. Further, these scenarios may be generated entirely by the TSSM or may be drawn from a preexisting scenario database and subjected to the same filtration process. After the scenarios assembled by the TSSM are executed, the methodology aggregates their driving assessment (DA) scores into a single numerical value. We outline the overall safety case-based framework, the TSSM, including its role in the framework as well as planned future work, and outline two proofs of concept: (1) a demonstration of the ability of the TSSM to pare down the space of scenarios in a scenario database; and (2) a specification form which may be used to solicit a description of the AV ODD and behavioral competency portfolio from the AV developer.

Original languageEnglish (US)
JournalSAE Technical Papers
StatePublished - Apr 9 2024
Event2024 SAE World Congress Experience, WCX 2024 - Detroit, United States
Duration: Apr 16 2024Apr 18 2024

ASJC Scopus subject areas

  • Automotive Engineering
  • Safety, Risk, Reliability and Quality
  • Pollution
  • Industrial and Manufacturing Engineering


Dive into the research topics of 'A Scenario-Based Test Selection and Scoring Methodology for Inclusion in a Safety Case Framework for Automated Vehicles'. Together they form a unique fingerprint.

Cite this