Improving Fault Localization for Simulink Models using Search-Based Testing and Prediction Models
0 likes253 views
This document summarizes an approach to improve fault localization for Simulink models using search-based testing and prediction models. The approach generates a small set of additional test cases to diversify an existing test suite using different test objectives. Predictor models are used to determine when adding more test cases is unlikely to improve fault localization rankings, allowing test generation to stop. An evaluation on 60 faulty industrial Simulink models found the approach significantly improved fault localization accuracy while reducing the number of new test cases generated by over half compared to generating test cases without the predictor models.
1 of 18
Download to read offline
![Test Case Generation
• We use a Single-State search technique to generate new test cases
• Our search strategy is guided by three alternative test objectives that aim
to increase the diversity of the test suite
• Dynamic basic blocks; Baudry et al. [ICSE’06]
• Coverage Density; Campos et al. [ASE’13]
• Coverage dissimilarity; Jiang et al. [ASE’09]
• None of the above test objectives require test oracles
11](https://image.slidesharecdn.com/saner2017bingslidesv5-170304154821/85/Improving-Fault-Localization-for-Simulink-Models-using-Search-Based-Testing-and-Prediction-Models-11-320.jpg)
![Results (1)
15
• [Accuracies of different test objectives]
ü The generated test cases significantly improved the fault localization accuracy
ü The three test objectives outperform Random test generation
ü No significant difference in the three test objectives](https://image.slidesharecdn.com/saner2017bingslidesv5-170304154821/85/Improving-Fault-Localization-for-Simulink-Models-using-Search-Based-Testing-and-Prediction-Models-15-320.jpg)
![Results (2)
16
• [Impact of adding test cases]
ü Adding test cases may not always improve fault localization accuracy](https://image.slidesharecdn.com/saner2017bingslidesv5-170304154821/85/Improving-Fault-Localization-for-Simulink-Models-using-Search-Based-Testing-and-Prediction-Models-16-320.jpg)
![Results (3)
17
• [Effectiveness of predictor model]
ü Our approach is able to maintain almost the same fault localization accuracy
while reducing the average number of newly generated test cases by more
than half.](https://image.slidesharecdn.com/saner2017bingslidesv5-170304154821/85/Improving-Fault-Localization-for-Simulink-Models-using-Search-Based-Testing-and-Prediction-Models-17-320.jpg)
Ad
Recommended
Extracting Domain Models from Natural-Language Requirements: Approach and Ind...
Extracting Domain Models from Natural-Language Requirements: Approach and Ind...Lionel Briand 1) The document presents an approach for automatically extracting domain models from natural language requirements documents. It uses grammatical dependencies and novel extraction rules to construct domain models from requirement statements.
2) An empirical evaluation of the approach on three case studies found that generic extraction rules were triggered most frequently, while pattern-based rules were seldom triggered. Interviews on one case study found the extracted relations to be highly correct on average but only moderately relevant.
3) The conclusions indicate that the novel extraction rules hold practical significance but room remains to improve the relevance of automatically extracted domain models, such as by addressing common knowledge and natural language processing mistakes.
Automated Test Suite Generation for Time-Continuous Simulink Models
Automated Test Suite Generation for Time-Continuous Simulink ModelsLionel Briand This document summarizes an approach for automated test suite generation for Simulink models with time-continuous behaviors. It discusses two main challenges with existing Simulink testing techniques: incompatibility with the underlying SAT/SMT-based techniques which cannot handle features like time-continuous blocks, and low fault revealing ability when test oracles are manual. The proposed approach uses search-based test generation driven by output diversity and failure patterns to generate test cases that are more likely to reveal faults. An evaluation compares the fault detection capability of the approach to Simulink Design Verifier and finds that the proposed output diversity technique outperforms it. The approach is implemented in a tool called SimCoTest.
Incremental Reconfiguration of Product Specific Use Case Models for Evolving ...
Incremental Reconfiguration of Product Specific Use Case Models for Evolving ...Lionel Briand 1. The document discusses an approach for incrementally reconfiguring product-specific use case models when configuration decisions evolve over time.
2. Product-specific models are regenerated by focusing only on the changed decisions and their side effects, preserving unaffected parts and existing trace links.
3. The approach involves matching decision model elements before and after changes, calculating differences, and using these to reconfigure use case diagrams and specifications while generating an impact report.
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Lionel Briand This document discusses scalable software testing and verification of non-functional properties through heuristic search and optimization. It describes several projects with industry partners that use metaheuristic search techniques like hill climbing and genetic algorithms to generate test cases for non-functional properties of complex, configurable software systems. The techniques address issues of scalability and practicality for engineers by using dimensionality reduction, surrogate modeling, and dynamically adjusting the search strategy in different regions of the input space. The results provided worst-case scenarios more effectively than random testing alone.
Testing the Untestable: Model Testing of Complex Software-Intensive Systems
Testing the Untestable: Model Testing of Complex Software-Intensive SystemsLionel Briand This document discusses model testing as an approach to testing complex, software-intensive systems that are difficult or impossible to fully automate. It presents model testing as shifting the focus of testing from implemented systems to executable models that capture relevant system behavior and properties. Model testing aims to find and execute high-risk test scenarios in large input spaces and help guide targeted testing of implemented systems. Challenges include defining testable models that include dynamic and uncertain behavior, performing effective test selection, and detecting failures under uncertainty.
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...
Evaluating Model Testing and Model Checking for Finding Requirements Violatio...Lionel Briand Model checking and model testing were compared for finding requirements violations in Simulink models. Model checking uses formal methods to exhaustively verify properties, but may not be applicable to some model constructs. Model testing uses meta-heuristic search to dynamically execute models and identify failures. Both approaches were applied to an industrial Simulink benchmark containing 11 models and 92 requirements. Model testing found violations of 40 requirements, while model checking could exhaustively prove 41 requirements and found violations of 23 requirements, but could not handle one of the models.
Testing Dynamic Behavior in Executable Software Models - Making Cyber-physica...
Testing Dynamic Behavior in Executable Software Models - Making Cyber-physica...Lionel Briand This document discusses testing dynamic behavior in executable software models for cyber-physical systems. It presents challenges for model-in-the-loop (MiL) testing due to large input spaces, expensive simulations, and lack of simple oracles. The document proposes using search-based testing to generate critical test cases by formulating it as a multi-objective optimization problem. It demonstrates the approach on an advanced driver assistance system and discusses improving performance with surrogate modeling.
System Testing of Timing Requirements based on Use Cases and Timed Automata
System Testing of Timing Requirements based on Use Cases and Timed AutomataLionel Briand The document describes a technique called TAUC that combines use case specifications and timed automata to automatically generate test cases for validating timing requirements of safety-critical systems. TAUC models the system functionality and environment as timed automata, identifies test inputs from use case scenarios to trigger state transitions, and employs a coverage-based and metaheuristic search approach to generate a test suite that stresses timing constraints. An evaluation on a case study shows TAUC achieves a 91% fault detection rate, significantly outperforming random and manual testing.
Testing of Cyber-Physical Systems: Diversity-driven Strategies
Testing of Cyber-Physical Systems: Diversity-driven StrategiesLionel Briand Lionel Briand discusses strategies for testing cyber-physical systems using diversity-driven approaches. He outlines challenges in verifying controllers and decision-making components in cyber-physical systems due to large input spaces and expensive model execution. Briand proposes maximizing diversity of test cases to improve fault detection. He describes using diversity of input signals, output signals, and failure patterns to generate test cases. Search algorithms are used to find test cases that maximize diversity or reveal specific failure patterns. The strategies are shown to significantly outperform coverage-based and random testing on Simulink models.
Analyzing Natural-Language Requirements: The Not-too-sexy and Yet Curiously D...
Analyzing Natural-Language Requirements: The Not-too-sexy and Yet Curiously D...Lionel Briand The document discusses challenges in analyzing natural language requirements and how natural language processing (NLP) techniques can help address these challenges. It describes challenges faced by industry such as ensuring compliance with templates, handling domain knowledge, enabling traceability and change impact analysis, and configuring requirements. It then discusses approaches developed through collaborative research to help with template conformance checking, change impact analysis between requirements, and analyzing impact of changes from requirements to design. The approaches leverage NLP techniques such as text chunking, syntactic and semantic analysis. Evaluation with industrial partners found the approaches to be effective at analyzing hundreds of requirements with high accuracy and limiting unnecessary inspection effort during change.
Applying Product Line Use Case Modeling ! in an Industrial Automotive Embedde...
Applying Product Line Use Case Modeling ! in an Industrial Automotive Embedde...Lionel Briand 1. The document describes a refined approach to product line use case modeling called PUM that was applied in an automotive embedded system project at IEE.
2. PUM models variability in use case diagrams, specifications, and domain models using extensions to existing modeling artifacts like use case diagrams and restricted use case modeling.
3. The approach aims to support change management and impact analysis in product lines while limiting additional modeling overhead.
Applications of Machine Learning and Metaheuristic Search to Security Testing
Applications of Machine Learning and Metaheuristic Search to Security TestingLionel Briand This document discusses testing web application firewalls (WAFs) for SQL injection (SQLi) vulnerabilities. It states that the testing goal is to generate test cases that result in executable malicious SQL statements that can bypass the WAF. It also notes that WAF filter rules often need customization to avoid false positives and protect against new attacks, but that customization is error-prone due to complex rules, time/resource constraints, and a lack of automated tools.
Search-Based Robustness Testing of Data Processing Systems
Search-Based Robustness Testing of Data Processing SystemsLionel Briand This document describes an approach to automatically generate robustness test cases for data processing systems using search-based testing and evolutionary algorithms. The approach uses model-based mutation to generate test inputs and model-based validation as an oracle to evaluate the test cases. An evolutionary algorithm is used to search for test cases that optimize coverage of various objectives like covering different parts of the data model, different types of data faults, clauses in input/output constraints, and code elements. The effectiveness of generated test suites is determined by measuring coverage of these objectives. The approach was evaluated on an industrial data processing system as a case study.
Documented Requirements are not Useless After All!
Documented Requirements are not Useless After All!Lionel Briand The document discusses challenges related to requirements in agile software development projects, including dealing with natural language requirements, domain knowledge, frequent changes, and configuring requirements for product families. It outlines research conducted to address these challenges through natural language processing techniques like analyzing compliance with boilerplate templates, extracting domain models from requirements, and analyzing change impact when requirements change. The talk aims to discuss when documented requirements are important in practice and how they can be supported to effectively handle common challenges.
Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
Effective Test Suites for ! Mixed Discrete-Continuous Stateflow ControllersLionel Briand The document describes algorithms for generating effective test suites for mixed discrete-continuous controllers modeled in Stateflow. It introduces the challenges of testing cyber-physical systems with both discrete and continuous behaviors. It then presents six test generation algorithms, including ones based on input diversity, state/transition coverage, and output diversity/stability/continuity. An evaluation of these algorithms on three industrial case studies examines their fault detection abilities, how they compare to each other, and how test suite size impacts results. The best performing algorithms focused on maximizing differences between output signals.
AN EMPIRICAL STUDY ON THE POTENTIAL USEFULNESS OF DOMAIN MODELS FOR COMPLETEN...
AN EMPIRICAL STUDY ON THE POTENTIAL USEFULNESS OF DOMAIN MODELS FOR COMPLETEN...Lionel Briand The study empirically examined the potential usefulness of domain models for detecting incompleteness in natural language requirements (RQ1). Domain models were found to provide useful hints towards requirements omissions, with higher sensitivity to unspecified versus underspecified requirements. Additionally, the sensitivity of domain models in detecting omissions varied based on the intrinsic properties of requirements documents, such as the frequency of key phrases (RQ2). Specifically, key phrases in requirements provided an accurate way to predict domain model sensitivity to omissions without full analysis. Therefore, domain models show promise as an external source for requirements completeness checking.
Testing Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveLionel Briand The document discusses the challenges and methodologies for testing machine learning (ML)-enabled systems, particularly in critical domains like automotive and healthcare. It highlights the complexities of model testing, including the need for realism in inputs, the relevance of adversarial and natural inputs, and the significance of online vs. offline testing. Various strategies such as automated emergency braking systems and multi-objective optimization are explored to enhance the robustness and effectiveness of ML testing practices.
PUMConf: A Tool to Configure Product Specific Use Case and Domain Models in a...
PUMConf: A Tool to Configure Product Specific Use Case and Domain Models in a...Lionel Briand The document describes PUMConf, a tool that helps configure product-specific use case and domain models from product line models. PUMConf allows modeling variability directly in use case diagrams, specifications, and domain models without using feature models. It provides automated consistency checking of product line models, interactive configuration support with consistency checking of decisions, and automated generation of product-specific models from configured product line models. An evaluation in an industrial case study found the approach and tool to be practical and beneficial for configuring product models in industrial settings.
SSBSE 2020 keynote
SSBSE 2020 keynoteShiva Nejati This document discusses search-based testing and its applications in software testing. It outlines some key strengths of search-based software testing (SBST) such as being scalable, parallelizable, versatile, and flexible. It also discusses some limitations of search-based approaches for problems that require formal verification to establish properties for all possible usages. The document compares classical optimization approaches, which build solutions incrementally, to stochastic optimization approaches used in SBST, which sample solutions in a randomized way. It notes that while testing can find bugs, it cannot prove their absence. Finally, it discusses how SBST can be combined with other techniques like constraint solving and machine learning.
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsLionel Briand This document discusses automated testing of autonomous driving assistance systems. It begins by introducing autonomous systems and their testing challenges due to large and complex input spaces and lack of explicit specifications. The document then describes an approach that combines evolutionary algorithms and decision tree classification models to guide testing towards critical scenarios. Evolutionary algorithms are used to search the input space while decision trees learn to predict scenario criticality and guide the search towards critical regions. The technique iteratively refines the decision tree model and focuses search on critical regions identified in the trees. The goal is to efficiently generate failure-revealing test cases and characterize input conditions that lead to critical situations.
Metamorphic Security Testing for Web Systems
Metamorphic Security Testing for Web SystemsLionel Briand Metamorphic testing is proposed to address the oracle problem in web security testing. Relations capture necessary properties between multiple inputs and outputs that must hold when a system is not vulnerable. Experiments on commercial and open source systems show the approach has high sensitivity (58.33-83.33%) and specificity (99.43-99.50%), detecting vulnerabilities without many false alarms. Extensive experiments with 22 relations achieved similar results for Jenkins and Joomla.
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...Lionel Briand The document describes a search-based testing approach for detecting XML injection vulnerabilities in web applications. The approach uses genetic algorithms to search the input space to generate malicious XML outputs defined as test objectives (TOs). The approach was evaluated on four subjects and found to be highly effective at detecting vulnerabilities, achieving 100% TO coverage. Random search was not effective, covering zero TOs. The approach was efficient, taking 5-32 minutes per TO. Input validation decreased coverage while fewer inputs and a restricted alphabet increased efficiency.
AI in SE: A 25-year Journey
AI in SE: A 25-year JourneyLionel Briand - The document discusses the speaker's 25 years of experience applying AI techniques to software engineering projects. It covers early work in the 1990s on fault prediction and the challenges of applying machine learning at that time. It then discusses subsequent work in areas like search-based software engineering, natural language processing for requirements engineering, and using simulation and search techniques for testing autonomous vehicle systems. The speaker reflects on both the benefits and challenges of these different AI applications in software engineering.
STAR: Stack Trace based Automatic Crash Reproduction
STAR: Stack Trace based Automatic Crash ReproductionSung Kim This document summarizes Ning Chen's PhD thesis defense on STAR, a stack trace based automatic crash reproduction framework. The outline includes motivation and related work, approaches of STAR including crash precondition computation and input model generation, evaluation study, challenges and future work, and contributions. STAR aims to efficiently reproduce crashes using only stack traces by performing backward symbolic execution to optimize crash precondition computation and address object creation challenges in reproducing object-oriented crashes.
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsLionel Briand This document discusses automated testing techniques for autonomous driving assistance systems (ADAS). It proposes using decision tree classification models and a multi-objective genetic search algorithm (NSGAII) to efficiently explore the complex scenario space of ADAS. The objectives are to identify critical, failure-revealing test scenarios by characterizing input conditions that lead to safety violations, such as the car hitting a pedestrian. Simulator-based testing of the automated emergency braking system is computationally expensive, so decision trees provide better guidance to the search by partitioning the input space into homogeneous regions.
Can we predict the quality of spectrum-based fault localization?
Can we predict the quality of spectrum-based fault localization?Lionel Briand The document discusses predicting the effectiveness of spectrum-based fault localization techniques. It proposes defining metrics to capture aspects of source code, test executions, test suites, and faults. A dataset of 341 instances with 70 variables is generated from Defects4J projects, classifying instances as "effective" or "ineffective" based on fault ranking. Analysis identifies the most influential metrics, finding a combination of static, dynamic, and test metrics can construct a prediction model with excellent discrimination, achieving an AUC of 0.86-0.88. The results suggest effectiveness depends more on code and test complexity than fault type/location, and entangled dynamic call graphs hinder localization.
Test Case Prioritization for Acceptance Testing of Cyber Physical Systems
Test Case Prioritization for Acceptance Testing of Cyber Physical SystemsLionel Briand 1) The document describes a multi-objective search-based approach for minimizing and prioritizing acceptance test cases for cyber physical systems to address challenges like time overhead, uncertainties in execution time, and risks of hardware damage.
2) The approach models acceptance tests, minimizes test cases by removing redundant operations, and prioritizes test cases using Monte Carlo simulation to estimate execution times while optimizing for criticality and risk.
3) An empirical evaluation on an industrial case study of in-orbit satellite testing shows the approach generates test suites that cover more test cases and have lower hardware risk within time budgets compared to manually created test suites.
HITECS: A UML Profile and Analysis Framework for Hardware-in-the-Loop Testing...
HITECS: A UML Profile and Analysis Framework for Hardware-in-the-Loop Testing...Lionel Briand This document describes HITECS, a UML profile and analysis framework for specifying and analyzing hardware-in-the-loop (HiL) test cases for cyber-physical systems. HITECS addresses challenges with HiL testing such as risks of hardware damage, time budget constraints, and environmental uncertainties. It provides a modeling language to specify test platforms, behaviors, analyses, and schedules. HITECS also supports model checking of test case assertions and simulation of test execution times to help evaluate test cases prior to execution. An empirical evaluation on a satellite testing case study found that HITECS helps engineers define effective assertions, verify test cases efficiently, and accurately estimate execution times.
Sistemas de información
Sistemas de informaciónLeonardo Quintero R El documento habla sobre los sistemas de información gerenciales (SIG) y los diferentes tipos de sistemas de información en las organizaciones. Explica que los SIG son sistemas integrados que optimizan la recopilación, transferencia y presentación de información a través de una organización, ayudando a los gerentes a planificar, organizar, dirigir y controlar. También describe los diferentes niveles donde operan los SIG y los tipos de informes que generan, así como los sistemas en el nivel operativo.
Programación CdA 5º de Educación Primaria
Programación CdA 5º de Educación PrimariaÁngel Alsasua Santos El proyecto implica que alumnos de quinto de primaria, organizados en grupos, actúan como meteorólogos para exponer sobre el tiempo y clima en España, incluyendo la predicción atmosférica y la vegetación asociada a diferentes climas. Se trabajará en diversas áreas como ciencias sociales, lengua extranjera y educación artística, enfatizando el uso del inglés en actividades prácticas como la realización de guiones y la creación de vídeos. La evaluación se basará en la cooperación del grupo, el uso de tecnologías, y la presentación de su trabajo mediante rúbricas que miden competencias específicas.
More Related Content
What's hot (20)
Testing of Cyber-Physical Systems: Diversity-driven Strategies
Testing of Cyber-Physical Systems: Diversity-driven StrategiesLionel Briand Lionel Briand discusses strategies for testing cyber-physical systems using diversity-driven approaches. He outlines challenges in verifying controllers and decision-making components in cyber-physical systems due to large input spaces and expensive model execution. Briand proposes maximizing diversity of test cases to improve fault detection. He describes using diversity of input signals, output signals, and failure patterns to generate test cases. Search algorithms are used to find test cases that maximize diversity or reveal specific failure patterns. The strategies are shown to significantly outperform coverage-based and random testing on Simulink models.
Analyzing Natural-Language Requirements: The Not-too-sexy and Yet Curiously D...
Analyzing Natural-Language Requirements: The Not-too-sexy and Yet Curiously D...Lionel Briand The document discusses challenges in analyzing natural language requirements and how natural language processing (NLP) techniques can help address these challenges. It describes challenges faced by industry such as ensuring compliance with templates, handling domain knowledge, enabling traceability and change impact analysis, and configuring requirements. It then discusses approaches developed through collaborative research to help with template conformance checking, change impact analysis between requirements, and analyzing impact of changes from requirements to design. The approaches leverage NLP techniques such as text chunking, syntactic and semantic analysis. Evaluation with industrial partners found the approaches to be effective at analyzing hundreds of requirements with high accuracy and limiting unnecessary inspection effort during change.
Applying Product Line Use Case Modeling ! in an Industrial Automotive Embedde...
Applying Product Line Use Case Modeling ! in an Industrial Automotive Embedde...Lionel Briand 1. The document describes a refined approach to product line use case modeling called PUM that was applied in an automotive embedded system project at IEE.
2. PUM models variability in use case diagrams, specifications, and domain models using extensions to existing modeling artifacts like use case diagrams and restricted use case modeling.
3. The approach aims to support change management and impact analysis in product lines while limiting additional modeling overhead.
Applications of Machine Learning and Metaheuristic Search to Security Testing
Applications of Machine Learning and Metaheuristic Search to Security TestingLionel Briand This document discusses testing web application firewalls (WAFs) for SQL injection (SQLi) vulnerabilities. It states that the testing goal is to generate test cases that result in executable malicious SQL statements that can bypass the WAF. It also notes that WAF filter rules often need customization to avoid false positives and protect against new attacks, but that customization is error-prone due to complex rules, time/resource constraints, and a lack of automated tools.
Search-Based Robustness Testing of Data Processing Systems
Search-Based Robustness Testing of Data Processing SystemsLionel Briand This document describes an approach to automatically generate robustness test cases for data processing systems using search-based testing and evolutionary algorithms. The approach uses model-based mutation to generate test inputs and model-based validation as an oracle to evaluate the test cases. An evolutionary algorithm is used to search for test cases that optimize coverage of various objectives like covering different parts of the data model, different types of data faults, clauses in input/output constraints, and code elements. The effectiveness of generated test suites is determined by measuring coverage of these objectives. The approach was evaluated on an industrial data processing system as a case study.
Documented Requirements are not Useless After All!
Documented Requirements are not Useless After All!Lionel Briand The document discusses challenges related to requirements in agile software development projects, including dealing with natural language requirements, domain knowledge, frequent changes, and configuring requirements for product families. It outlines research conducted to address these challenges through natural language processing techniques like analyzing compliance with boilerplate templates, extracting domain models from requirements, and analyzing change impact when requirements change. The talk aims to discuss when documented requirements are important in practice and how they can be supported to effectively handle common challenges.
Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers
Effective Test Suites for ! Mixed Discrete-Continuous Stateflow ControllersLionel Briand The document describes algorithms for generating effective test suites for mixed discrete-continuous controllers modeled in Stateflow. It introduces the challenges of testing cyber-physical systems with both discrete and continuous behaviors. It then presents six test generation algorithms, including ones based on input diversity, state/transition coverage, and output diversity/stability/continuity. An evaluation of these algorithms on three industrial case studies examines their fault detection abilities, how they compare to each other, and how test suite size impacts results. The best performing algorithms focused on maximizing differences between output signals.
AN EMPIRICAL STUDY ON THE POTENTIAL USEFULNESS OF DOMAIN MODELS FOR COMPLETEN...
AN EMPIRICAL STUDY ON THE POTENTIAL USEFULNESS OF DOMAIN MODELS FOR COMPLETEN...Lionel Briand The study empirically examined the potential usefulness of domain models for detecting incompleteness in natural language requirements (RQ1). Domain models were found to provide useful hints towards requirements omissions, with higher sensitivity to unspecified versus underspecified requirements. Additionally, the sensitivity of domain models in detecting omissions varied based on the intrinsic properties of requirements documents, such as the frequency of key phrases (RQ2). Specifically, key phrases in requirements provided an accurate way to predict domain model sensitivity to omissions without full analysis. Therefore, domain models show promise as an external source for requirements completeness checking.
Testing Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveLionel Briand The document discusses the challenges and methodologies for testing machine learning (ML)-enabled systems, particularly in critical domains like automotive and healthcare. It highlights the complexities of model testing, including the need for realism in inputs, the relevance of adversarial and natural inputs, and the significance of online vs. offline testing. Various strategies such as automated emergency braking systems and multi-objective optimization are explored to enhance the robustness and effectiveness of ML testing practices.
PUMConf: A Tool to Configure Product Specific Use Case and Domain Models in a...
PUMConf: A Tool to Configure Product Specific Use Case and Domain Models in a...Lionel Briand The document describes PUMConf, a tool that helps configure product-specific use case and domain models from product line models. PUMConf allows modeling variability directly in use case diagrams, specifications, and domain models without using feature models. It provides automated consistency checking of product line models, interactive configuration support with consistency checking of decisions, and automated generation of product-specific models from configured product line models. An evaluation in an industrial case study found the approach and tool to be practical and beneficial for configuring product models in industrial settings.
SSBSE 2020 keynote
SSBSE 2020 keynoteShiva Nejati This document discusses search-based testing and its applications in software testing. It outlines some key strengths of search-based software testing (SBST) such as being scalable, parallelizable, versatile, and flexible. It also discusses some limitations of search-based approaches for problems that require formal verification to establish properties for all possible usages. The document compares classical optimization approaches, which build solutions incrementally, to stochastic optimization approaches used in SBST, which sample solutions in a randomized way. It notes that while testing can find bugs, it cannot prove their absence. Finally, it discusses how SBST can be combined with other techniques like constraint solving and machine learning.
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsLionel Briand This document discusses automated testing of autonomous driving assistance systems. It begins by introducing autonomous systems and their testing challenges due to large and complex input spaces and lack of explicit specifications. The document then describes an approach that combines evolutionary algorithms and decision tree classification models to guide testing towards critical scenarios. Evolutionary algorithms are used to search the input space while decision trees learn to predict scenario criticality and guide the search towards critical regions. The technique iteratively refines the decision tree model and focuses search on critical regions identified in the trees. The goal is to efficiently generate failure-revealing test cases and characterize input conditions that lead to critical situations.
Metamorphic Security Testing for Web Systems
Metamorphic Security Testing for Web SystemsLionel Briand Metamorphic testing is proposed to address the oracle problem in web security testing. Relations capture necessary properties between multiple inputs and outputs that must hold when a system is not vulnerable. Experiments on commercial and open source systems show the approach has high sensitivity (58.33-83.33%) and specificity (99.43-99.50%), detecting vulnerabilities without many false alarms. Extensive experiments with 22 relations achieved similar results for Jenkins and Joomla.
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...
A Search-based Testing Approach for XML Injection Vulnerabilities in Web Appl...Lionel Briand The document describes a search-based testing approach for detecting XML injection vulnerabilities in web applications. The approach uses genetic algorithms to search the input space to generate malicious XML outputs defined as test objectives (TOs). The approach was evaluated on four subjects and found to be highly effective at detecting vulnerabilities, achieving 100% TO coverage. Random search was not effective, covering zero TOs. The approach was efficient, taking 5-32 minutes per TO. Input validation decreased coverage while fewer inputs and a restricted alphabet increased efficiency.
AI in SE: A 25-year Journey
AI in SE: A 25-year JourneyLionel Briand - The document discusses the speaker's 25 years of experience applying AI techniques to software engineering projects. It covers early work in the 1990s on fault prediction and the challenges of applying machine learning at that time. It then discusses subsequent work in areas like search-based software engineering, natural language processing for requirements engineering, and using simulation and search techniques for testing autonomous vehicle systems. The speaker reflects on both the benefits and challenges of these different AI applications in software engineering.
STAR: Stack Trace based Automatic Crash Reproduction
STAR: Stack Trace based Automatic Crash ReproductionSung Kim This document summarizes Ning Chen's PhD thesis defense on STAR, a stack trace based automatic crash reproduction framework. The outline includes motivation and related work, approaches of STAR including crash precondition computation and input model generation, evaluation study, challenges and future work, and contributions. STAR aims to efficiently reproduce crashes using only stack traces by performing backward symbolic execution to optimize crash precondition computation and address object creation challenges in reproducing object-oriented crashes.
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsLionel Briand This document discusses automated testing techniques for autonomous driving assistance systems (ADAS). It proposes using decision tree classification models and a multi-objective genetic search algorithm (NSGAII) to efficiently explore the complex scenario space of ADAS. The objectives are to identify critical, failure-revealing test scenarios by characterizing input conditions that lead to safety violations, such as the car hitting a pedestrian. Simulator-based testing of the automated emergency braking system is computationally expensive, so decision trees provide better guidance to the search by partitioning the input space into homogeneous regions.
Can we predict the quality of spectrum-based fault localization?
Can we predict the quality of spectrum-based fault localization?Lionel Briand The document discusses predicting the effectiveness of spectrum-based fault localization techniques. It proposes defining metrics to capture aspects of source code, test executions, test suites, and faults. A dataset of 341 instances with 70 variables is generated from Defects4J projects, classifying instances as "effective" or "ineffective" based on fault ranking. Analysis identifies the most influential metrics, finding a combination of static, dynamic, and test metrics can construct a prediction model with excellent discrimination, achieving an AUC of 0.86-0.88. The results suggest effectiveness depends more on code and test complexity than fault type/location, and entangled dynamic call graphs hinder localization.
Test Case Prioritization for Acceptance Testing of Cyber Physical Systems
Test Case Prioritization for Acceptance Testing of Cyber Physical SystemsLionel Briand 1) The document describes a multi-objective search-based approach for minimizing and prioritizing acceptance test cases for cyber physical systems to address challenges like time overhead, uncertainties in execution time, and risks of hardware damage.
2) The approach models acceptance tests, minimizes test cases by removing redundant operations, and prioritizes test cases using Monte Carlo simulation to estimate execution times while optimizing for criticality and risk.
3) An empirical evaluation on an industrial case study of in-orbit satellite testing shows the approach generates test suites that cover more test cases and have lower hardware risk within time budgets compared to manually created test suites.
HITECS: A UML Profile and Analysis Framework for Hardware-in-the-Loop Testing...
HITECS: A UML Profile and Analysis Framework for Hardware-in-the-Loop Testing...Lionel Briand This document describes HITECS, a UML profile and analysis framework for specifying and analyzing hardware-in-the-loop (HiL) test cases for cyber-physical systems. HITECS addresses challenges with HiL testing such as risks of hardware damage, time budget constraints, and environmental uncertainties. It provides a modeling language to specify test platforms, behaviors, analyses, and schedules. HITECS also supports model checking of test case assertions and simulation of test execution times to help evaluate test cases prior to execution. An empirical evaluation on a satellite testing case study found that HITECS helps engineers define effective assertions, verify test cases efficiently, and accurately estimate execution times.
Viewers also liked (15)
Sistemas de información
Sistemas de informaciónLeonardo Quintero R El documento habla sobre los sistemas de información gerenciales (SIG) y los diferentes tipos de sistemas de información en las organizaciones. Explica que los SIG son sistemas integrados que optimizan la recopilación, transferencia y presentación de información a través de una organización, ayudando a los gerentes a planificar, organizar, dirigir y controlar. También describe los diferentes niveles donde operan los SIG y los tipos de informes que generan, así como los sistemas en el nivel operativo.
Programación CdA 5º de Educación Primaria
Programación CdA 5º de Educación PrimariaÁngel Alsasua Santos El proyecto implica que alumnos de quinto de primaria, organizados en grupos, actúan como meteorólogos para exponer sobre el tiempo y clima en España, incluyendo la predicción atmosférica y la vegetación asociada a diferentes climas. Se trabajará en diversas áreas como ciencias sociales, lengua extranjera y educación artística, enfatizando el uso del inglés en actividades prácticas como la realización de guiones y la creación de vídeos. La evaluación se basará en la cooperación del grupo, el uso de tecnologías, y la presentación de su trabajo mediante rúbricas que miden competencias específicas.
Guía de aprendizaje 5 slideshare
Guía de aprendizaje 5 slidesharemaria valeria torres moncada Este documento presenta una guía de aprendizaje sobre la introducción a SlideShare. Explica que SlideShare es un sitio web para compartir presentaciones y documentos. Los usuarios pueden subir, ver y compartir archivos como presentaciones de PowerPoint, PDF y documentos de Word. También permite transmitir presentaciones en vivo y guardar diapositivas individuales. La guía incluye instrucciones para registrarse en SlideShare, cargar y descargar archivos, e insertar videos de YouTube en una presentación.
Faq häufig gestellte fragen
Faq häufig gestellte fragenUlrich Kritzler Die Kieferorthopädie befasst sich mit der Diagnose und Behandlung von Fehlstellungen der Zähne und Kiefer, um die Lebensqualität der Patienten zu verbessern. Die erste Untersuchung sollte idealerweise zwischen 5 und 7 Jahren erfolgen, auch bei funktionellen Störungen, und die Behandlungen können bei Erwachsenen mit speziellen Systemen wie Alignern ebenso durchgeführt werden. Ein Fachzahnarzt für Kieferorthopädie ist notwendig für die richtige Diagnose und Therapieplanung, da er eine umfassende Ausbildung in diesem Bereich hat.
Canal de Panamá y Universidad
Canal de Panamá y Universidaddocenciagrupo02 El documento describe el papel del Instituto del Canal de Panamá y Estudios Internacionales (ICUP) de la Universidad de Panamá en investigar y analizar los principales problemas que enfrenta Panamá en el orden internacional, como el futuro del Canal de Panamá. El ICUP ha creado programas de maestría relacionados con el Canal y ha publicado estudios sobre la identidad nacional de Panamá y sus relaciones con Estados Unidos. También ha establecido convenios de cooperación con otras instituciones para fomentar la investigación en temas relacionados con el
04_Node modules
04_Node modulesAhmed Elbassel The document outlines the concept of modules in Node.js, explaining their purpose for organizing and reusing code. It provides a guide on creating, importing, and using parameters in modules, along with exercises for practical application, including a simple calculator. A bonus exercise offers additional modifications to enhance user interaction.
Contexto Artistico e Cultural
Contexto Artistico e CulturalEduardo Becker Jr. O documento discute a importância da arte na educação. A arte ajuda os alunos a desenvolverem sua sensibilidade, percepção e imaginação. Um professor que não teve experiência com diferentes formas de arte não saberá observar e promover a criatividade dos alunos nessas áreas. A arte capacita as pessoas a compreenderem e transformarem a realidade de forma mais humana.
Dynamic Website Personalization - Make your website more interactive.
Dynamic Website Personalization - Make your website more interactive.LeadGen-IMPACT The document discusses dynamic website personalization (DWP), emphasizing its role in tailoring website experiences to individual visitors based on various factors. It highlights the importance of leveraging customer knowledge for enhanced engagement and increased sales while providing examples of effective implementations. Additionally, it underscores the need for ongoing content analysis and optimization to maximize customer conversion rates.
Clasificacion de tornillos
Clasificacion de tornillosalexisadonay El documento describe los diferentes tipos de tornillos y sus designaciones. Explica que la designación de un tornillo incluye el tipo de cabeza, la rosca, la longitud y la norma. Luego detalla los diferentes tipos de roscas según normas como DIN, ISO y BS, así como métodos de seguridad como contratuercas, arandelas elásticas y pasadores.
Resins
ResinsSubha Sri Aala The document provides an extensive overview of plant resins, including their definitions, properties, occurrences, and distributions. Key sections discuss methods of extraction, classifications, and various uses of specific resins in pharmaceuticals and other applications. It also includes detailed information about different plants producing resins, their cultivation, and chemical constituents.
Clase sociales 5°-03-06-17_plan lector 01_ficha 02
Clase sociales 5°-03-06-17_plan lector 01_ficha 02IETI AJC Olga Lucía Lloreda Este documento instruye al lector a elaborar una segunda ficha de lectura resumiendo los derechos fundamentales descritos en la Constitución Política de Colombia usando una herramienta como una tabla, gráfico, Paint o CmapTools. También pide al lector que investigue e ilustre la organización del estado colombiano.
Clase inglés 5°_03-09-17_check my notes
Clase inglés 5°_03-09-17_check my notesIETI AJC Olga Lucía Lloreda The document outlines a review of class notes from an English course taught by William S. Díaz in Cali on March 9, 2017. It includes learning objectives and activities related to reviewing notes and completing assignments. Additionally, there is a homework assignment to create a card for a close female friend.
Industrial engineering – case study
Industrial engineering – case studyJephin S John This document discusses a case study on optimizing manpower usage through time study methods at Mahindra HINO Motors. The objectives were to minimize excessive manpower, production time, and efforts while improving production methods and ergonomics. Through time study analysis of operations like raw material unloading, inspection, numbering, and loading, solutions like hydraulic lifters and pneumatic numbering machines were implemented. This reduced the total time for a cycle from 270 seconds to 60 seconds, minimizing manpower requirements by 2-3 people per cycle while meeting objectives.
Clase castellano 5°-03-09-17_revisión_apuntes_y_actividades
Clase castellano 5°-03-09-17_revisión_apuntes_y_actividadesIETI AJC Olga Lucía Lloreda El documento presenta el objetivo de revisar los apuntes y actividades de clase del curso de Castellano 5° hasta la fecha para asegurar que estén al día y completos. También propone comparar los apuntes y trabajos con compañeros de grupo y desarrollar una autoevaluación para identificar cualquier tema o actividad que falte por cubrir o completar, así como resumir las reglas de uso de mayúsculas en un cuadro como tarea.
Ad
Similar to Improving Fault Localization for Simulink Models using Search-Based Testing and Prediction Models (20)
Making Model-Driven Verification Practical and Scalable: Experiences and Less...
Making Model-Driven Verification Practical and Scalable: Experiences and Less...Lionel Briand The document discusses experiences and lessons learned from making model-driven verification practical and scalable. It describes several projects collaborating with industry partners to develop model-based solutions for verification. Key challenges addressed include achieving applicability for engineers, scalability to large systems, and developing solutions informed by real-world problems. Lessons learned emphasize the importance of collaborative applied research, defining problems in context, and validating solutions realistically.
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceLionel Briand 1. The document discusses using artificial intelligence techniques like machine learning and natural language processing to help automate software testing. It focuses on applying these techniques to testing advanced driver assistance systems.
2. A key challenge in software testing is scalability as the input spaces and code bases grow large and complex. Effective automation is needed to address this challenge. The document describes several industrial research projects applying AI to help automate testing of advanced driver assistance systems.
3. One project aims to develop an automated testing technique for emergency braking systems in cars using a physics-based simulation. The goal is to efficiently explore complex test scenarios and identify critical situations like failures to avoid collisions.
Achieving Scalability in Software Testing with Machine Learning and Metaheuri...
Achieving Scalability in Software Testing with Machine Learning and Metaheuri...Lionel Briand This document discusses challenges in testing advanced driver assistance systems (ADAS) and approaches to address scalability. It describes using physics-based simulation and search-based testing to generate test cases for an automated emergency braking system. The testing faces challenges due to the large, complex input space and computational expense of simulations. Decision trees are proposed to better guide the search by partitioning the input space into homogeneous regions based on criticality.
SBST 2019 Keynote
SBST 2019 Keynote Shiva Nejati The document summarizes research conducted by the Software Verification and Validation Group at the University of Luxembourg on testing cyber physical systems via evolutionary algorithms and machine learning. The group develops techniques to generate test inputs for autonomous systems like automated driving using genetic algorithms and optimizes the search process using machine learning. This guided search aims to find test cases that could reveal violations of critical safety requirements with fewer simulation runs. The approach is demonstrated on an industrial project that tests requirements for the automated emergency braking features of autonomous vehicles.
An Approach to Software Testing of Machine Learning Applications
An Approach to Software Testing of Machine Learning Applicationsbutest This document describes an approach to software testing of machine learning applications. The approach involves analyzing the problem domain, algorithm, and implementation options to generate test cases. Two case studies applying this approach found bugs in implementations of MartiRank and SVM ranking algorithms. Analyzing the problem domain uncovered issues like how to handle missing/negative values. Analyzing algorithms revealed potential specification imprecisions. Analyzing options showed how inputs could be manipulated. The approach helped find bugs, create regression test cases, and validate implementations by comparing results across different versions and ML algorithms.
Wavelet Based on the Finding of Hard and Soft Faults in Analog and Digital Si...
Wavelet Based on the Finding of Hard and Soft Faults in Analog and Digital Si...ijcisjournal This paper presents methods for detecting hard and soft faults in analog and digital signal circuits using wavelet transform techniques. It introduces two algorithms based on Euclidean and Mahalanobis distances for analyzing wavelet energy, with the aim of improving fault detection effectiveness in practical applications. The study highlights the challenges of fault simulation in analog circuits and emphasizes the importance of reliable testing methodologies for ensuring quality in integrated circuits.
Logic Simulation, Modeling, and Testing
Logic Simulation, Modeling, and TestingAksum Institute of Technology(AIT, @Letsgo) The document discusses simulation, modeling, and testing in VLSI design. It covers various topics including logic simulation, fault simulation, and VLSI testing. Logic simulation verifies design correctness using simulation. Fault simulation measures test effectiveness by simulating faults. VLSI testing verifies manufactured chips using test generation and application. The document compares different simulation and testing techniques.
How to find defects early and increase the reliability of software systems
How to find defects early and increase the reliability of software systemsRAKESH RANA The document discusses strategies for early defect detection and increased reliability in automotive software systems, focusing on fault injection and mutation testing techniques. It highlights the importance of combining these methods with fault bypass modeling to effectively identify safety defects during the development phase. The proposed framework aims to enhance ISO 26262 compliance and improve overall software robustness, ultimately reducing development costs and time to market.
A simplified predictive framework for cost evaluation to fault assessment usi...
A simplified predictive framework for cost evaluation to fault assessment usi...IJECEIAES The paper presents a novel predictive framework for software fault evaluation using machine learning methodologies to improve cost assessment and fault prediction accuracy. It introduces a correlation-based cost model integrated with machine learning approaches, including linear regression, support vector regression, and artificial neural networks, focusing on factors such as training loss and imbalanced data. Experimental results demonstrate that the neural network model effectively outperforms traditional methods in fault prediction accuracy.
50120140501001
50120140501001IAEME Publication This document summarizes an article from the International Journal of Computer Engineering and Technology. The article proposes a reliability improvement predictive approach to software testing using mathematical modeling and the Empirical Bayesian method. It introduces a model predictive control framework for software testing. The key aspects are:
1) It uses the Empirical Bayesian method to estimate reliability and optimize the test allocation scheme online by repeatedly solving an optimal control problem.
2) A case study shows that the proposed approach can achieve better results in improving reliability than random testing.
3) The case study also demonstrates that finding more defects through testing does not necessarily lead to higher reliability, and the proposed approach more effectively directs testing efforts towards the most important bugs.
Ijsea04031006
Ijsea04031006Editor IJCATR The document discusses a model-based test case prioritization approach for regression testing using genetic algorithms, aiming to enhance testing efficiency and reduce costs. It contrasts code-based and model-based prioritization methods, emphasizing the advantages of model-based techniques in detecting severe faults earlier. The study highlights a genetic algorithm's role in generating effective test sequences, ultimately leading to cost savings and improved software quality.
Automated Testing of Hybrid Simulink/Stateflow Controllers
Automated Testing of Hybrid Simulink/Stateflow ControllersLionel Briand This document discusses automated testing of Simulink/Stateflow controllers for automotive software. It presents an approach using black box search-based testing to generate test cases for closed-loop and open-loop controllers. The approach uses fitness functions to evaluate test results and provide failure explanation and detection. Case studies on industrial models demonstrate generating test inputs that reveal failures and visualizing the input space to explain under what conditions failures are likely to occur. The approach aims to help engineers test complex models without manual oracles or dealing with tool incompatibilities.
Lionel Briand ICSM 2011 Keynote
Lionel Briand ICSM 2011 KeynoteICSM 2011 The document discusses the challenges and gaps in software engineering research, highlighting a disconnect between academic efforts and practical industry needs. It emphasizes the necessity for academic research to be problem-driven and to involve real-world applications, advocating for collaboration between academia and industry. The piece concludes that improving the relevance and impact of software engineering research is crucial for the profession's future and calls for a collective effort from all stakeholders involved.
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...
st-notes-13-26-software-testing-is-the-act-of-examining-the-artifacts-and-the...mwpeexdvjgtqujwhog The document discusses software testing and hardware testing. Some key differences are:
- Software faults remain unless the application is changed, while hardware can fail over time due to new faults.
- Software testing examines artifacts and behavior to detect present faults, while hardware testing can apply built-in self tests rarely used for software.
- Test domains differ between hardware bit patterns and software tuples of data types.
A survey of fault prediction using machine learning algorithms
A survey of fault prediction using machine learning algorithmsAhmed Magdy Ezzeldin, MSc. The document surveys four machine learning approaches for predicting software faults, emphasizing the importance of software reliability and fault prediction. It discusses methods including fuzzy logic, support vector machines, genetic algorithms, and comparative analysis of machine learning algorithms, highlighting their effectiveness and accuracy in identifying fault-prone software components. The findings suggest that machine learning can significantly aid in early fault prediction, potentially reducing project risks and costs.
Presentation Verification & Validation
Presentation Verification & ValidationElmar Selbach This document discusses requirements traceability and testing capabilities in Simulink. It describes how Simulink can trace requirements bidirectionally between models and source code. It also outlines how Simulink Design Verifier can automatically generate tests to achieve high coverage and check designs against requirements. Finally, it positions several MathWorks products in the V&V process and highlights key capabilities of Simulink Test for authoring, managing and executing simulation-based tests.
Mining Assumptions for Software Components using Machine Learning
Mining Assumptions for Software Components using Machine LearningLionel Briand EPIcuRus is an approach to automatically generate assumptions for software components in cyber-physical systems modeled in Simulink. It uses machine learning techniques to mine assumptions from test case results. The approach includes generating tests cases using an important feature boundary test generation method, model checking candidate assumptions, and selecting the most informative safe assumptions. An evaluation on industrial case studies found it can learn non-vacuous assumptions for most requirements within a practical time limit, with the important feature boundary test generation performing best.
VLSI testing and analysis
VLSI testing and analysisSurekha PuriGosavi This document discusses VLSI testing and analysis. It defines key terms like defect, fault, and error and describes typical types of defects. It also discusses logical fault models and the role of testing in quality control. Different types of tests like production testing and burn-in testing are described. The testing process, fault simulation, design for testability techniques, and built-in self-test are summarized.
Presentation
PresentationSATYALOK Software testing is important for quality assurance and finding errors. However, complete testing is infeasible due to the complexity of software. There are many types of testing like black-box, white-box, correctness, and performance testing. Testing should be automated when possible to improve efficiency and effectiveness. It is difficult to determine when to stop testing but metrics like reliability and budgets can provide guidelines. While testing has limitations, it remains a crucial part of the software development process.
Ad
More from Lionel Briand (20)
LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on...
LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on...Lionel Briand FSE 2025 presentation
TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural N...
TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural N...Lionel Briand FSE 2025 presentation
Automated Testing and Safety Analysis of Deep Neural Networks
Automated Testing and Safety Analysis of Deep Neural NetworksLionel Briand Keynote address at Internetware 2025, co-located with FSE 2025
FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categorie...
FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categorie...Lionel Briand Journal-first presentation at ICST 2025
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Lionel Briand Keynote at RAISE workshop, ICSE 2025
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalLionel Briand The document discusses the challenges of achieving precise and complete requirements upfront in software development projects. It notes that while academics assume detailed requirements are needed, practitioners find this difficult to achieve in reality due to limited resources, uncertainty, and changing needs. The document provides perspectives from practice that emphasize starting with prototypes and visions rather than detailed specifications. It also summarizes research finding diverse requirements practices across different domains and organizations. The document concludes that while precise requirements may be desirable, they are often elusive goals, and the focus should be on achieving compliance and delivering working software.
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand Large language models show promise for test case repair tasks. LLMs can be applied to tasks like test case generation, classification of flaky tests, and test case evolution and repair. The paper presents TaRGet, a framework that uses LLMs for automated test case repair. TaRGet takes as input a broken test case and code changes to the system under test, and outputs a repaired test case. Evaluation shows TaRGet achieves over 80% plausible repair accuracy. The paper analyzes repair characteristics, evaluates different LLM and input/output formats, and examines the impact of fine-tuning data size on performance.
Metamorphic Testing for Web System Security
Metamorphic Testing for Web System SecurityLionel Briand This document summarizes a presentation on metamorphic testing for web system security given by Nazanin Bayati on September 13, 2023. Metamorphic testing uses relations between the outputs of multiple test executions to test systems when specifying expected outputs is difficult. It was applied to web systems by generating follow-up inputs based on transformations of valid interactions and checking that output relations held. The approach detected over 60% of vulnerabilities in tested systems and addressed more vulnerability types than static and dynamic analysis tools. It provides an effective and automated way to test for security issues in web systems.
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...Lionel Briand This document proposes a method called SEDE (Simulator-based Explanations for DNN Errors) to automatically generate explanations for errors in DNN-based safety-critical systems by constraining simulator parameters. SEDE first identifies clusters of error-inducing images, then uses an evolutionary algorithm to generate simulator images within each cluster, including failing, passing, and representative images. SEDE extracts rules characterizing the unsafe parameter space and uses the generated images to retrain DNNs, improving accuracy compared to alternative methods. The paper evaluates SEDE on head pose and face landmark detection DNNs in terms of generating diverse cluster images, delimiting unsafe spaces, and enhancing DNN performance.
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingLionel Briand This document summarizes a research paper on using grey-box fuzzing (MOTIF) for mutation testing of C/C++ code in cyber-physical systems (CPS). It introduces mutation testing and grey-box fuzzing, and proposes MOTIF which generates a fuzzing driver to test functions with live mutants. An empirical evaluation compares MOTIF to symbolic execution-based mutation testing on three subject programs. MOTIF killed more mutants within 10,000 seconds and was able to test programs that symbolic execution could not handle due to limitations like floating-point values. Seed inputs alone killed few mutants, showing the importance of fuzzing. MOTIF is an effective approach for mutation testing of CPS software.
Data-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical SystemsLionel Briand Data-driven mutation analysis is proposed to assess if test suites for cyber-physical systems properly exercise component interoperability. Fault models are developed for different data types and dependencies, and are used to automatically generate mutants by injecting faults. Empirical results on industrial systems demonstrate the feasibility and effectiveness of the approach in identifying test suite shortcomings and poor oracles.
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled SystemsLionel Briand This document proposes MORLOT (Many-Objective Reinforcement Learning for Online Testing) to address challenges in online testing of DNN-enabled systems. MORLOT leverages many-objective search and reinforcement learning to choose test actions. It was evaluated on the Transfuser autonomous driving system in the CARLA simulator using 6 safety requirements. MORLOT was significantly more effective and efficient at finding safety violations than random search or other many-objective approaches, achieving a higher average test effectiveness for any given test budget.
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...Lionel Briand 1. The document presents ATM, a new approach for black-box test case minimization that transforms test code into abstract syntax trees and uses tree-based similarity measures and genetic algorithms to minimize test suites.
2. ATM was evaluated on the DEFECTS4J dataset and achieved a fault detection rate of 0.82 on average, significantly outperforming existing techniques, while requiring only practical execution times.
3. The best configuration of ATM used a genetic algorithm with a combined similarity measure, achieving a fault detection rate of 0.80 within 1.2 hours on average.
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...Lionel Briand The document is a journal paper that proposes a method for black-box safety analysis and retraining of deep neural networks (DNNs) based on feature extraction and clustering of failure-inducing images. The method uses a pre-trained VGG16 model to extract features from failure images, clusters the features using DBSCAN, selects clusters that likely caused failures, and retrains the DNN to improve safety based on images in problematic clusters. An empirical evaluation on various DNNs for tasks like gaze detection showed the method effectively determined failure causes through clustering and improved models with fewer images than other approaches.
PRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System LogsLionel Briand PRINS is a technique for scalable model inference of component-based system logs. It divides the problem into inferring individual component models and then stitching them together. The paper evaluates PRINS on several systems and compares its execution time and accuracy to MINT, a state-of-the-art model inference tool. Results show that PRINS is significantly faster than MINT, especially on larger logs, with comparable accuracy. However, stitching component models can result in larger overall system models. The paper contributes an empirical evaluation of the PRINS technique and makes its implementation publicly available.
Revisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software TestingLionel Briand The document discusses the concept of diversity in software testing. It provides examples of how diversity has been applied in various testing applications, including test case prioritization and minimization, mutation analysis, and explaining errors in deep neural networks. The key aspects of diversity discussed are the representation of test cases, measures of distance or similarity between cases, and techniques for maximizing diversity. The document emphasizes that the best approach depends on factors like information access, execution costs, and the specific application context.
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...Lionel Briand This document discusses search-based approaches for testing artificial intelligence systems. It covers testing at different levels, from model-level testing of individual machine learning components to system-level testing of AI-enabled systems. At the model level, search-based techniques are used to generate test inputs that target weaknesses in deep learning models. At the system level, simulations and reinforcement learning are used to test AI components integrated into complex systems. The document outlines many open challenges in AI testing and argues that search-based approaches are well-suited to address challenges due to the complex, non-linear behaviors of AI systems.
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and SafetyLionel Briand Autonomous systems present safety challenges due to their complexity and use of machine learning. Two key approaches are needed to address these challenges: (1) design-time assurance cases to validate safety requirements and (2) run-time monitoring architectures to detect unsafe behavior. Automated testing techniques leveraging metaheuristics and machine learning can help provide evidence for assurance cases and learn conditions to guide run-time monitoring. However, more industrial experience is still needed to properly validate these approaches at scale for autonomous systems.
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...Lionel Briand This document discusses the split identities of software engineering researchers between being mathematicians, social scientists, or engineers. It notes there are three main communities - formal methods and guarantees, human and social studies, and engineering automated solutions - that have different backgrounds, languages, and research methods. While diversity is good, the communities need to be better connected to work together to solve problems. The document calls for more demand-driven, collaborative research with industry to have a greater impact and produce practical solutions.
Recently uploaded (20)
Advance Doctor Appointment Booking App With Online Payment
Advance Doctor Appointment Booking App With Online PaymentAxisTechnolabs https://www.slideshare.net/slideshow/advance-doctor-appointment-booking-app-with-online-payment/280654375
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso The pdf-version of "CodeCleaner: Mitigating Data Contamination for LLM Benchmarking" presented in Internetware 2025.
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI TechIFI Techsolutions Start your cloud journey with IFI Tech—Microsoft’s trusted partner delivering secure, scalable Azure solutions tailored for your business.
From Data Preparation to Inference: How Alluxio Speeds Up AI
From Data Preparation to Inference: How Alluxio Speeds Up AIAlluxio, Inc. Alluxio Webinar
June 17, 2025
For more Alluxio Events: https://www.alluxio.io/events/
Speaker:
Jingwen Ouyang (Sr. Product Manager @ Alluxio)
In this talk, Jingwen Ouyang, Senior Product Manager at Alluxio, will share how Alluxio make it easy to share and manage data from any storage to any compute engine in any environment with high performance and low cost for your model training, model inference, and model distribution workload.
Top Time Tracking Solutions for Accountants
Top Time Tracking Solutions for Accountantsoliviareed320 This presentation explores the importance of time tracking for accountants and introduces modern solutions that help streamline billable hours, improve productivity, and eliminate manual tracking errors. Learn how time tracking software tailored for accounting professionals can simplify client billing, ensure compliance, and enhance profitability.
From key challenges in time management to top features to look for in a solution, this presentation offers valuable insights and practical tips for solo accountants, finance teams, and accounting firms looking to optimize their time-tracking workflows.https://www.invoicera.com/blog/invoicing/best-time-tracking-software-for-accountants/
Download Adobe Illustrator Crack free for Windows 2025?
Download Adobe Illustrator Crack free for Windows 2025?grete1122g COPY & PASTE LINK 👉👉👉 https://click4pc.com/after-verification-click-go-to-download-page/
Adobe Illustrator CC Crack will make your illustration level upgrade by moving toward real modification purpose using adobe pro tools there ..
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...Philip Schwarz For newer version see https://fpilluminated.org/deck/264
In this deck we look at the following:
* How unfolding lists is the computational dual of folding lists
* Different variants of the function for unfolding lists
* How they relate to the iterate function
Y - Recursion The Hard Way GopherCon EU 2025
Y - Recursion The Hard Way GopherCon EU 2025Eleanor McHugh In most modern programming languages, including Go, it's possible for a function to call itself recursively. It's such a mainstream technique that we've come to take it for granted.
But what if functions couldn't call themselves?
There's a branch of mathematics called Combinatorics which answers just these kinds of questions, and this presentation takes these ideas and phrases them in Go to teach a deeper view of computation through the medium of code.
Key Challenges in Troubleshooting Customer On-Premise Applications
Key Challenges in Troubleshooting Customer On-Premise ApplicationsTier1 app When applications are deployed in customer-managed environments, resolving performance issues becomes a hidden battle. Limited access to logs, delayed responses, and vague problem descriptions make root cause analysis incredibly challenging.
In this presentation, we walk through:
The six most common hurdles faced in on-prem troubleshooting
A practical 360° artifact collection strategy using the open-source yc-360 script
Real-world case studies covering transaction timeouts, CPU spikes, and intermittent HTTP 502 errors
Benefits of structured data capture in reducing investigation time and improving communication with customers
✅ Whether you're supporting enterprise clients or building tools for distributed environments, this deck offers a practical roadmap to make on-prem issue resolution faster and less frustrating.
Why Edge Computing Matters in Mobile Application Tech.pdf
Why Edge Computing Matters in Mobile Application Tech.pdfIMG Global Infotech This PDF explores the critical role of edge computing in modern mobile application technology. Learn how edge enhances app speed, reduces latency, improves data privacy, and supports real-time performance. Ideal for developers, tech strategists, and mobile innovators looking to stay ahead in the evolving mobile ecosystem.
Building Geospatial Data Warehouse for GIS by GIS with FME
Building Geospatial Data Warehouse for GIS by GIS with FMESafe Software Data warehouses are often considered the backbone of data-driven decision-making. However, traditional implementations can struggle to meet the iterative and dynamic demands of GIS operations, particularly for Iterative Cross-Division Data Integration (ICDI) processes. At WVDOT, we faced this challenge head-on by designing a Geospatial Data Warehouse (GDW) tailored to the needs of GIS. In this talk, we’ll share how WVDOT developed a metadata-driven, SESSI-based (Simplicity, Expandability, Spatiotemporality, Scalability, Inclusivity) GDW that transforms how we manage and leverage spatial data. Key highlights include: - Simplifying data management with a lean schema structure. - Supporting ad hoc source data variations using JSON for flexibility. - Scaling for performance with RDBMS partitioning and materialized views. - Empowering users with FME Flow-based self-service metadata tools.We’ll showcase real-world use cases, such as pavement condition data and LRS-based reporting, and discuss how this approach is elevating GIS as an integrative framework across divisions. Attendees will walk away with actionable insights on designing GIS-centric data systems that combine the best of GIS and data warehouse methodologies.
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso The pdf version of "CodeCleaner: Mitigating Data Contamination for LLM Benchmarking" presented in Internetware 2025.
declaration of Variables and constants.pptx
declaration of Variables and constants.pptxmeemee7378 Rules of declaration of variables and constants
OpenChain Webinar - AboutCode - Practical Compliance in One Stack – Licensing...
OpenChain Webinar - AboutCode - Practical Compliance in One Stack – Licensing...Shane Coughlan OpenChain Webinar - AboutCode - Practical Compliance in One Stack – Licensing, Vulnerabilities, and More
Introduction to Agile Frameworks for Product Managers.pdf
Introduction to Agile Frameworks for Product Managers.pdfAli Vahed As a Product Manager, having a solid understanding of Agile frameworks is essential, whether you are joining an established team or helping shape a new one.
Agile is not just a delivery method; it is a mind set that directly impacts how products are built, iterated, and brought to market. Knowing the strengths and trade-offs of each framework enables you to collaborate more effectively with teams, foster alignment, and drive meaningful outcomes.
Simply put, Agile fluency helps you lead with clarity in fast-paced, ever-changing environments.
The next few slides are simple ‘STUDY CARDS’ to help you learn or refresh your understanding of a few popular Agile frameworks.
How Automation in Claims Handling Streamlined Operations
How Automation in Claims Handling Streamlined OperationsInsurance Tech Services Claims processing automation reduces manual work, cuts errors, and speeds up resolutions. Using AI, RPA, and OCR, it enables faster data handling and smoother workflows—boosting efficiency and customer satisfaction. Visit https://www.damcogroup.com/insurance/claims-processing-automation for more details!
Why Every Growing Business Needs a Staff Augmentation Company IN USA.pdf
Why Every Growing Business Needs a Staff Augmentation Company IN USA.pdfmary rojas U.S. staff augmentation companies excel at providing IT professionals with expertise in development, cybersecurity, AI, and more. Whether you need to fill a short-term gap or enhance your team’s capabilities long-term, these firms deliver pre-vetted talent tailored to your tech stack and industry. With an understanding of both global technologies and local compliance, U.S.-based IT staff augmentation bridges the gap between talent shortage and project demand efficiently.
Foundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025
Foundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025BradBedford3 Join us for an exciting introductory session on, Foundations of Marketo Engage: Programs, Campaigns & Beyond. Ideal for new and early-stage users looking to build confidence and capability in Marketo Engage.
Our speakers for this session will be:
AJ Navarro
Marketing Operations Manager - Sprout Social
Bobby Coppola
Marketing Manager - Blue Yonder
This event will guide you through the essential steps to build strong, scalable Programs, Channels, Tags, Smart Campaigns, and Reporting Basics in Marketo Engage.
Attendees will gain practical knowledge on how to structure and launch marketing initiatives, leverage templates for efficiency, and implement best practices for campaign execution and measurement. You'll leave with actionable insights you can immediately apply to your own Marketo instance to improve organization, automation, and reporting. Marketo Engage.
Learn how to create impactful programs and understand the best practices for seamless campaign management and effective reporting.
Don’t miss out on this opportunity to gain hands-on guidance from experienced practitioners and to connect with peers who are also new to the platform. Secure your spot today and start building your path to becoming a Marketo pro!
Please feel free to invite colleagues who you think would also benefit from this session.
Best MLM Compensation Plans for Network Marketing Success in 2025
Best MLM Compensation Plans for Network Marketing Success in 2025LETSCMS Pvt. Ltd.
Discover the top MLM compensation plans including Unilevel, Binary, Matrix, Board, and Australian Plans. Learn how to choose the best plan for your business growth with expert insights from MLM Trees. Explore hybrid models, payout strategies, and earning potential.
Learn more: https://www.mlmtrees.com/mlm-plans/
From Code to Commerce, a Backend Java Developer's Galactic Journey into Ecomm...
From Code to Commerce, a Backend Java Developer's Galactic Journey into Ecomm...Jamie Coleman In a galaxy not so far away, a Java developer advocate embarks on an epic quest into the vast universe of e-commerce. Armed with backend languages and the wisdom of microservice architectures, set out with us to learn the ways of the Force. Navigate the asteroid fields of available tools and platforms; tackle the challenge of integrating location-based technologies into open-source projects; placate the Sith Lords by enabling great customer experiences.
Follow me on this journey from humble Java coder to digital marketplace expert. Through tales of triumph and tribulation, gain valuable insights into conquering the e-commerce frontier - such as the different open-source solutions available - and learn how technology can bring balance to the business Force, large or small. May the code be with you.
Improving Fault Localization for Simulink Models using Search-Based Testing and Prediction Models
- 1. .lusoftware verification & validation VVS Improving Fault Localization for Simulink Models using Search-Based Testing and Prediction Models Bing Liu, Lucia, Shiva Nejati, Lionel Briand SnT Centre, University of Luxembourg SANER 2017, Klagenfurt, Austria
- 2. Simulation 2
- 3. Simulink • Is a data flow-driven block diagram language • Is widely used in the automotive domain • Is executable and enables simulation and early testing • Supports automated code generation 3
- 5. Background: Statistical Debugging 5 b4 b2 b3 b5 b7b6 b8 b10 b11 b12 b14b13 b9 b1 Block Rankings b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 b13 b14 input1 input2 input3 output
- 6. 6 Ranking Score b1 0 b2 0 b3 0 b4 0 b5 0 b6 0 b7 0 b8 0 b9 0 b10 0 b11 0 b12 0 b13 0 b14 0 input1 input2 input3 output Execute test case 1 b4 b2 b3 b5 b7b6 b8 b10 b11 b12 b14b13 b9 b1 Background: Statistical Debugging
- 7. 7 Ranking Score b6 1 b2 0.5 b3 0.5 b5 0.5 b7 0.5 b8 0.5 b9 0.5 b10 0.5 b12 0.5 b13 0.5 b14 0.5 b1 0 b4 0 b11 0 input1 input2 input3 output Execute test case 2 b4 b2 b3 b5 b7b6 b8 b10 b11 b12 b14b13 b9 b1 Background: Statistical Debugging
- 8. Problem: Performance Limitations 8 Ranking Score b6 1 b2 0.5 b3 0.5 b5 0.5 b7 0.5 b8 0.5 b9 0.5 b10 0.5 b12 0.5 b13 0.5 b14 0.5 b1 0 b4 0 b11 0 • Faulty blocks may not be ranked high • Many blocks may have the same score • Engineers may have to inspect many blocks until they find the faulty block(s)
- 9. Goal: Improving Statistical Debugging • Statistical debugging can be improved by using larger test suites • But, adding test cases is not cost-free in some contexts because * test oracles need to be developed manually * running test cases might be expensive • We need to generate a small but effective set of test cases 9
- 10. Our Approach 10 Model Test Suite + Test Oracle Ranking Generate a small but effective set of test cases No YesRegenerate rankings Are existing rankings likely to be improved by adding more test cases? Static analysis + Predictor models By diversifying test cases
- 11. Test Case Generation • We use a Single-State search technique to generate new test cases • Our search strategy is guided by three alternative test objectives that aim to increase the diversity of the test suite • Dynamic basic blocks; Baudry et al. [ICSE’06] • Coverage Density; Campos et al. [ASE’13] • Coverage dissimilarity; Jiang et al. [ASE’09] • None of the above test objectives require test oracles 11
- 12. Stop Test Generation Criteria • We stop the test generation when adding test cases is unlikely to improve the rankings • We rely on * Simulink Super Blocks: based on static analysis of Simulink models * Predictor models built using a supervised learning technique and historical data 12
- 13. Decision Trees • Input features: • Current Round index • Set Distance • Ordering Distance 13
- 14. Experiment Evaluation • We applied our approach to 60 single-fault-seeded versions of our three industrial Simulink models • We compared our three different test objectives • We evaluated the effectiveness of predictor models • The experiment results (except for original industry models) are available at https://github.com/Avartar/TCGenForFL/) 14
- 15. Results (1) 15 • [Accuracies of different test objectives] ü The generated test cases significantly improved the fault localization accuracy ü The three test objectives outperform Random test generation ü No significant difference in the three test objectives
- 16. Results (2) 16 • [Impact of adding test cases] ü Adding test cases may not always improve fault localization accuracy
- 17. Results (3) 17 • [Effectiveness of predictor model] ü Our approach is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half.
- 18. Conclusion • We proposed a new approach to improve fault localization accuracy for Simulink models by extending an existing test suite with a small number of high quality test cases. • We evaluated our approach on 60 faulty versions of industry models. • Our approach significantly improves the accuracy of fault localization for small test suite sizes • Our approach is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half 18