Academic Journal
Toward Standardized Benchmarking of Search-Based Scenario Selection Methods in Autonomous System Validation.
| Τίτλος: | Toward Standardized Benchmarking of Search-Based Scenario Selection Methods in Autonomous System Validation. |
|---|---|
| Συγγραφείς: | Autenrieth, Daniel, Paland, Daniel, Mentjes, Jan, Zlocki, Adrian |
| Πηγή: | Journal of Marine Science & Engineering; Aug2026, Vol. 14 Issue 15, p1386, 39p |
| Θεματικοί όροι: | Benchmark testing (Engineering), Benchmarking (Management), Safety, Autonomous robots, Simulation software, Computer software testing |
| Περίληψη: | Autonomous systems must demonstrate reliable safety and robustness even under rare, safety-critical conditions. Because the vast range of possible operating situations cannot be exhaustively tested, scenario-based testing has emerged as a structured approach to expose systems to representative and challenging situations. Search-based scenario selection methods (SBSSMs) algorithmically explore the scenario space to identify critical and informative cases. Yet, existing implementations are often tightly coupled to specific domains and simulation environments, hindering reproducibility, comparability, and generalizability across application areas. This paper introduces a benchmarking framework that addresses these limitations by providing a domain-independent environment for the systematic evaluation of SBSSMs. Building on empirical observations from maritime simulation studies and the literature, the framework generates synthetic test instances that mimic the structural patterns found there while abstracting away application-specific semantics. It supplies standardized performance metrics and reference implementations of key method classes, enabling direct comparison of new approaches with established ones under controlled, repeatable conditions. Illustrative demonstrations show how the benchmark reveals characteristic strengths and weaknesses of different methods across diverse structural settings. As a reproducible and extensible baseline, the framework promotes standardized performance assessment and the transfer of methodological advances from maritime navigation to domains such as automotive safety. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Marine Science & Engineering is the property of MDPI and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Βάση Δεδομένων: | Complementary Index |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:edb&genre=article&issn=20771312&ISBN=&volume=14&issue=15&date=20260801&spage=1386&pages=1386-1424&title=Journal of Marine Science & Engineering&atitle=Toward%20Standardized%20Benchmarking%20of%20Search-Based%20Scenario%20Selection%20Methods%20in%20Autonomous%20System%20Validation.&aulast=Autenrieth%2C%20Daniel&id=DOI:10.3390/jmse14151386 Name: Full Text Finder (for New FTF UI) (ns324271) Category: fullText Text: Full Text Finder MouseOverText: Full Text Finder |
|---|---|
| Header | DbId: edb DbLabel: Complementary Index An: 196302478 RelevancyScore: 1082 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1082.42456054688 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Toward Standardized Benchmarking of Search-Based Scenario Selection Methods in Autonomous System Validation. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Autenrieth%2C+Daniel%22">Autenrieth, Daniel</searchLink><br /><searchLink fieldCode="AR" term="%22Paland%2C+Daniel%22">Paland, Daniel</searchLink><br /><searchLink fieldCode="AR" term="%22Mentjes%2C+Jan%22">Mentjes, Jan</searchLink><br /><searchLink fieldCode="AR" term="%22Zlocki%2C+Adrian%22">Zlocki, Adrian</searchLink> – Name: TitleSource Label: Source Group: Src Data: Journal of Marine Science & Engineering; Aug2026, Vol. 14 Issue 15, p1386, 39p – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Benchmark+testing+%28Engineering%29%22">Benchmark testing (Engineering)</searchLink><br /><searchLink fieldCode="DE" term="%22Benchmarking+%28Management%29%22">Benchmarking (Management)</searchLink><br /><searchLink fieldCode="DE" term="%22Safety%22">Safety</searchLink><br /><searchLink fieldCode="DE" term="%22Autonomous+robots%22">Autonomous robots</searchLink><br /><searchLink fieldCode="DE" term="%22Simulation+software%22">Simulation software</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+software+testing%22">Computer software testing</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Autonomous systems must demonstrate reliable safety and robustness even under rare, safety-critical conditions. Because the vast range of possible operating situations cannot be exhaustively tested, scenario-based testing has emerged as a structured approach to expose systems to representative and challenging situations. Search-based scenario selection methods (SBSSMs) algorithmically explore the scenario space to identify critical and informative cases. Yet, existing implementations are often tightly coupled to specific domains and simulation environments, hindering reproducibility, comparability, and generalizability across application areas. This paper introduces a benchmarking framework that addresses these limitations by providing a domain-independent environment for the systematic evaluation of SBSSMs. Building on empirical observations from maritime simulation studies and the literature, the framework generates synthetic test instances that mimic the structural patterns found there while abstracting away application-specific semantics. It supplies standardized performance metrics and reference implementations of key method classes, enabling direct comparison of new approaches with established ones under controlled, repeatable conditions. Illustrative demonstrations show how the benchmark reveals characteristic strengths and weaknesses of different methods across diverse structural settings. As a reproducible and extensible baseline, the framework promotes standardized performance assessment and the transfer of methodological advances from maritime navigation to domains such as automotive safety. [ABSTRACT FROM AUTHOR] – Name: Abstract Label: Group: Ab Data: <i>Copyright of Journal of Marine Science & Engineering is the property of MDPI and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edb&AN=196302478 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.3390/jmse14151386 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 39 StartPage: 1386 Subjects: – SubjectFull: Benchmark testing (Engineering) Type: general – SubjectFull: Benchmarking (Management) Type: general – SubjectFull: Safety Type: general – SubjectFull: Autonomous robots Type: general – SubjectFull: Simulation software Type: general – SubjectFull: Computer software testing Type: general Titles: – TitleFull: Toward Standardized Benchmarking of Search-Based Scenario Selection Methods in Autonomous System Validation. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Autenrieth, Daniel – PersonEntity: Name: NameFull: Paland, Daniel – PersonEntity: Name: NameFull: Mentjes, Jan – PersonEntity: Name: NameFull: Zlocki, Adrian IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 08 Text: Aug2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 20771312 Numbering: – Type: volume Value: 14 – Type: issue Value: 15 Titles: – TitleFull: Journal of Marine Science & Engineering Type: main |
| ResultId | 1 |