Revisiting the Notion of Diversity in Software Testing
0 likes269 views
The document discusses the concept of diversity in software testing. It provides examples of how diversity has been applied in various testing applications, including test case prioritization and minimization, mutation analysis, and explaining errors in deep neural networks. The key aspects of diversity discussed are the representation of test cases, measures of distance or similarity between cases, and techniques for maximizing diversity. The document emphasizes that the best approach depends on factors like information access, execution costs, and the specific application context.
Revisiting the Notion of Diversity in Software Testing
- 1. Revisiting the Notion of Diversity in Software Testing Lionel Briand SBFT 2023 Keynote http://www.lbriand.info
- 2. Why Diversity? âą Diverse test cases âą Exercising the system to the largest extent possible within a budget âą Increase probability of fault detection âą While working with incomplete knowledge âą Cost of acquiring information âą Missing information 2
- 3. Example: Fuzzing with AFL 3 Diversity mechanisms: Mutation, coverage Credits: Antonio Morales, https://github.com/antonio-morales/Fuzzing101
- 4. Aspects of Diversity 4 SUT Inputs Outputs Execution (internal): - Structural coverage - Model coverage (e.g., states)
- 5. Questions ⹠What aspects of diversity to focus on? ⹠Information access ⹠Information cost, e.g., execution time ⹠Context-dependent ⹠How to measure diversity? ⹠Representation (e.g., inputs) ⹠Distance measure, e.g., cosine, edit ⹠Computational cost ⹠Guidance, e.g., in search ⹠How to maximize diversity? ⹠Mutation, metaheuristic search, symbolic execution ⊠⹠Issues: cost, scalability, bias, effectiveness 5
- 6. Aspects of Diversity âą Inputs: No instrumentation, does not require the execution of the SUT âą Outputs: No instrumentation, execution required but directly characterizes the behavior of the SUT âą Internal SUT structure: Instrumentation, possibly modeling, additional execution cost and significant data storage 6
- 7. Example: Testing DNNs âą Redundant or invalid inputs âą Labeling cost is high âą Domain-specific knowledge is required to manually label test inputs âą Cost of test execution can be high âą Coverage ineffective âą Test selection based on inputs 7 Aghababaeyan et al., 2023
- 8. Example: Testing DNNs We want to test a DNN model with a fixed test budget. âą How can we automatically select a candidate test subset with high-fault revealing power to test DNNs? âą Black-box test selection based on input diversity. 8 Black-box test selection method Test inputs T Subset SâT
- 9. Example: Testing DNNs âą No model execution âą No access to model internals or training set âą Studies show that proposed coverage measures for DNNs not associated with faults âą Solution: Geometric diversity of image features 9
- 10. Extracting Image Features âą VGG16 is a convolutional neural network trained on a subset of the ImageNet dataset, a collection of over 14 million images belonging to 22,000 categories. 10 Features: - Activation values after last convolutional layer - Characterize semantic elements such as shapes and colors
- 11. Geometric Diversity (GD) âą Given a dataset X and its corresponding feature vectors V, the geometric diversity of a subset S â X is defined as the hyper-volume of the parallelepiped spanned by the rows of Vs, i.e., feature vectors of items in S, where the larger the volume, the more diverse is the feature space of S 11 Aghababaeyan et al., 2023
- 12. Measuring Diversity âą Representation and measure: Construct validity? âą Cost of computing diversity âą Guidance provided by diversity, e.g., test selection search 12
- 13. Example: Test Minimization âą Permanently remove redundant test cases in a test suite that are unlikely to detect new faults âą Black-box versus white-box techniques âą FAST-R: Quick and black-box, but low fault detection rates âą ATM: Abstract Syntax Tree (AST)-based Test case Minimizer âą Motivation: Achieve a better trade-off between effectiveness and efficiency than FAST-R âą Context: Minimization only applied to major releases 13
- 14. Example: ATM âą Representation: AST of pre-processed test code âą Tree similarity measures: top-down, bottom-up, combined, edit distance âą Common subtree isomorphism algorithms âą Top-down and bottom-up emphasize different aspects of similarity between ASTs 14 Transform test code to ASTs Test Suite Measure test case similarity Run search algorithms Minimized test suite Pre-process test code 4 tree-based similarity measures GA & NSGA-II Pan et al., 2023
- 15. Example: ATM âą Alternatives evaluated in terms of Fault Detection Rate (FDR) âą Edit distance is expensive but offers good guidance âą Combined similarity not significantly different âą Multi-objective search more expensive âą Much higher fault detection than FAST-R in significantly higher execution time, though practical up to an extent 15 GA NSGA-II Top-Down Bottom-Up Combined Tree Edit Distance Top-Down & Bottom-Up Combined & Tree Edit Distance FDR 0.78 Time 70.87 FDR 0.74 Time 67.05 FDR 0.80 Time 72.75 FDR 0.81 Time 82.23 FDR 0.78 Time 235.41 FDR 0.82 Time 258.44
- 16. Example: Input Diversity in DNNs âą Alternative diversity measures: Geometrics Diversity, Normalized Compression Distance (NCD), standard deviation âą Construct validity? âą Analysis: âą We study how diversity scores change while varying the number of classes or concepts inside the images of the input sets. âą We assume that diversity scores should increase with the number of classes or concepts that are present in an input set. 16
- 17. Example: Input Diversity in DNNs âą Geometrics Diversity shows a clear monotonic relationship with the number of classes in the input set 17 11 (a) Evolution of GD on Cifar-10 (b) Evolution of STD on Cifar-10 (c) Evolution of NCD on Cifar-10 (d) Evolution of GD on MNIST (e) Evolution of STD on MNIST (f) Evolution of NCD on MNIST Figure 8: Evolution of the diversity scores for input sets from Cifar-10 and MNIST. Each boxplot shows the distribution of diversity scores of 20 input sets of size 100. This article has been accepted for publication in IEEE Transactions on Software Engineering. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TSE.2023.3243522 Aghababaeyan et al., 2023
- 18. Applications âą Test minimization, selection, prioritization âą Mutation analysis âą Identify boundaries in the input space, e.g., safe vs unsafe 18
- 19. MASS: CPS Mutation Testing 19 Create mutants Compile mutants Killed Mutants Live Mutants 2 Collect test data 1 Code Coverage Remove equivalent/duplicate based on compiler optimizations 4 3 Mutants Code coverage Mutants successfully compiled Unique mutants Evaluate mutation scoreâs confidence Sampled mutants Sample mutants Execute prioritized subset of test cases 5 6 7 Cornejo et al., 2021 âą Selection and prioritization of test cases based on statement coverage âą Test suite prioritization: âą Greedy algorithm âą Select first the test case that largely differ from the most similar, already selected, test case âą Test suite reduction: exclude test cases with perfect similarity
- 20. MASS: CPS Mutation Testing âą Compare the sets of source code statements that have been covered by test cases: Jaccard and Ochiai âą Compare the number of times each statement has been covered by test cases: Euclidian, cosine âą Focus on functions in source file where mutated statement located âą Best: Cosine distance âą Difference in mutation score < 5% 20 Create mutants Compile mutants Killed Mutants Live Mutants 2 Collect test data 1 Code Coverage Remove equivalent/duplicate based on compiler optimizations 4 3 Mutants Code coverage Mutants successfully compiled Unique mutants Evaluate mutation scoreâs confidence Sampled mutants Sample mutants Execute prioritized subset of test cases 5 6 7 Reduction in mutation analysis time > 70%
- 21. Explanations for DNN Errors (SEDE) Can we explain DNN failures of real-world images using simulator parameters? 21 Training Set Simulator Images DNN Training Test Set Simulator Images DNN Testing DNN Training (fine-tuning) Training Set Real-world Images DNN Testing Trained DNN Fine-Tuned DNN Real-world Error Inducing Images Test Set Real-world Images
- 22. SEDE 22 Real-world Error-inducing images HUDD Evolutionary Algorithms Simulator Simulator images Configuration Parameters RCC Prototype Images Step 1. Identify root-cause clusters (RCCs) Step 2. Generate images associated to RCCs RCCs Step 2.1. Identify RCC Prototype Images Step 2.2. Generate a set of unsafe images belonging to the cluster Step 2.3. Generate one safe image for each unsafe image PaiR Error-inducing Test Set images Step1. Heatmap based clustering Root cause clusters C1 C2 C3 Step 2. Inspection of subset of cluster elements. HUDD: Fahmy et al. 2021 Cluster 2 (near closed eyes) incomplete training set Cluster 1 (angle ~157.5) borderline cases
- 23. SEDE 23 Synthetic images Parameters-based Description Improved DNN model Retraining: +18.6% Real-World Images HeadPosex > 10 & HeadPosey > 50.34 Real-world images Diverse simulator images, within the cluster Diverse failing simulator images, close to these images Passing simulator images, close to failing ones in cluster S1 S2 S3 Cluster Process: Simulator-based Explanations for DNN Errors (SEDE) Real-world Error-inducing images HUDD Evolutionary Algorithms Simulator Simulator images Configuration Parameters RCC Prototype Images Step 1. Identify root-cause clusters (RCCs) Step 2. Generate images associated to RCCs RCCs Step 2.1. Identify RCC Prototype Images Step 2.2. Generate a set of unsafe images belonging to the cluster Step 2.3. Generate one safe image for each unsafe image PaiR Fahmy et al. 2022
- 24. WCET for Critical Tasks 24 âą Real-time systems âą Schedulability analysis verifies time constraints for critical tasks âą Early schedulability analysis and design decisions require early task Worst Case Execution Time (WCET) estimates âą Challenges: Tasks not fully implemented, worst case inputs unknown âą Goal: Estimating Probabilistic Safe WCET Ranges at Design Stages
- 25. SAFE: WCET boundaries âą Safe WCET boundaries: implementation objectives, evaluate design options âą Iterative, distance-based sampling of WCET values within ranges 25 Phase 1. Worst-case task arrivals analysis Phase 2. Safe WCET computation Training dataset Worst-case sequences of task arrivals Task descriptions Search Learning Safe Unsafe WCET T1 WCET T2 SAFE: Safe WCET Analysis method For real-time task schEdulability (Lee et al., 2022)
- 26. Use of Language Models âą Code (e.g., test) or trace vector representation (encoding) âą Benefit from pre-trained language models, e.g., CodeBERT âą Example: test case prioritization (Test2Vec, Jabbar et al., 2022), minimization âą Embedding test execution traces with fine-tuned CodeBERT âą Fined-tuned with pass/fail labels for past test cases in a system âą Test2Vec maps test execution traces, i.e., sequences of method calls with their inputs and return values, to fixed-length, numerical vectors âą Heuristic: Similarity to previous failing test cases in the same project 26
- 28. Conclusions âą Many applications of diversity in testing âą Various aspects warrant different solutions: information access, execution cost, instrumentation cost âą Trade-off between representations (test cases) and distance/similarity measures: computation cost, guidance âą Determining the best solution can only be done empirically, in a well-defined (application) context âą Check assumptions and properties of distance/similarity measures, e.g., desired sensitivity to change âą Scalability is usually the stumbling block for many applications 28
- 29. Selected References âą Cornejo et al., âMutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domainâ, IEEE Transactions on Software Engineering, 2021 âą Fahmy et al. "Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning" IEEE Transactions on Reliability, Special section on Quality Assurance of Machine Learning Systems, 2021 âą Fahmy et al. "Simulator-based explanation and debugging of hazard-triggering events in DNN-based safety-critical systemsâ, ACM TOSEM, 2022 âą Attaoui et al., âBlack-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clusteringâ, ACM TOSEM, 2022 âą Pan et al., , âATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Searchâ, IEEE /ACM ICSE 2023, âą Lee et al., âEstimating probabilistic safe wcet ranges of real-time systems at design stagesâ, ACM TOSEM 2022 âą Jabbar et al., Test2Vec: An Execution Trace Embedding for Test Case Prioritization, ArXIV, 2022 âą Aghababaeyan et al., âBlack-Box Testing of Deep Neural Networks through Test Case Diversityâ, IEEE Transactions on Software Engineering, 2023 âą Aghababaeyan et al., âDeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networksâ, https://arxiv.org/abs/2303.04878 29
- 30. Looking for Postdocs! Lionel Briand SBFT 2023 Keynote http://www.lbriand.info