TMPA-2015: Towards a Usable Defect Prediction Tool: Crossbreeding Machine Learning and Heuristics
3 likes3,189 views
The document discusses the development of a defect prediction tool that combines machine learning and heuristics to identify defect-prone code entities efficiently. It highlights the shortcomings of existing tools in terms of accuracy and setup effort, while proposing usability criteria for future solutions. The results indicate that while combining approaches can improve predictions, the overall accuracy remains insufficient for widespread industry adoption.
TMPA-2015: Towards a Usable Defect Prediction Tool: Crossbreeding Machine Learning and Heuristics
- 1. Towards a usable defect prediction tool Vladimir Kovalenko JetBrains Crossbreeding machine learning and heuristics
- 2. Defect Prediction • Common goal: identify defect-prone entities in advance • Why? • QA • Resource allocation (testing, review, etc)
- 3. Previous research • Academia • Microsoft Research • Google case study
- 4. Research papers (common points) • ML defect prediction models work in general • No universal model: projects are too different • Code metrics as features improve prediction quality • Typical precision/recall ~0.7
- 5. Google case study • Collaborated with researchers to introduce defect prediction in internal code review system • Came up with a heuristic model Google Bug Prediction Score (Time Weighted Risk)
- 6. Tools • No defect prediction tools known to be used in industry • Why? • Too low accuracy • Too much effort to set up
- 7. Tool usability criteria • Language independent • “entity” >= file • Little or no effort to set up • no plain supervised learning • Near real-time • Easy to use: VCS agnostic, etc • Accurate!
- 8. Implementation • CI server plugin • Only use VCS metrics • Automatic bugfix changes detection (heuristics) • Processing: detect bug-introducing changes • ML classifier: Naive Bayes / Decision Tree • Take prediction top, not absolute values • Automatic quality evaluation
- 9. Features • Local change frequency • Number of authors • File age • Number of affecting commits • Google Score
- 10. Quality Evaluation • Bug tracker integration: find bugfix changes • Quality metric: fraction of files from model predictions affected by bugfix changes in the future
- 11. Result samples Project A, 2 years Project B, 1 year
- 12. Conclusions • It is possible combine learning and heuristic approaches to get the best of two worlds • The accuracy is still not good enough • No wonder no prediction tools are widely used yet