Domain-specific implications of error-type metrics in risk-based software fault prediction

Phung, Khoa; Ogunshile, Emmanuel; Aydin, Mehmet E.

Abstract

In software development, Software Fault Prediction (SFP) is essential for optimising resource allocation and improving testing efficiency. Traditional SFP methods typically use binary-class models, which can provide a limited perspective on the varying risk levels associated with individual software modules. This study explores the impacts of Error-type Metrics on the fault-proneness of software modules in domain-specific software projects. Also, it aims to enhance SFP methods by introducing a risk-based approach using Error-type Metrics. This method categorises software modules into High, Medium, and Low-Risk categories, offering a more granular and informative fault prediction framework. This approach aims to refine the fault prediction process and contribute to more effective resource allocation and project management in software development. We explore the domain-specific impact of Error-type Metrics through Principal Component Analysis (PCA), aiming to fill a gap in the existing literature by offering insights into how these metrics affect machine learning models across different software domains. We employ three machine learning models - Support Vector Machine (SVM), Random Forest (RF), and Extreme Gradient Boosting (XGB) - to test our approach. The Synthetic Minority Over-sampling Technique (SMOTE) is used to address class imbalance. Our methodology is validated on fault data from four open-source software projects, aiming to confirm the robustness and generalisability of our approach. The PCA findings provide evidence of the varied impacts of Error-type Metrics in different software environments. Comparative analysis indicates a strong performance by the XGB model, achieving an accuracy of 97.4%, a Matthews Correlation Coefficient of 96.1%, and an F1-score of 97.4% across the datasets. These results suggest the potential of the proposed method to contribute to software testing and quality assurance practices. Our risk-based SFP approach introduces a new perspective to risk assessment in software development. The study’s findings contribute insights into the domain-specific applicability of Error-type Metrics, expanding their potential utility in SFP. Future research directions include refining our fault-counting methodology and exploring broader applications of Error-type Metrics and our proposed risk-based approach.

• Journal Article Type: Article
• Acceptance Date: Nov 27, 2024
• Online Publication Date: Jan 7, 2025
• Publication Date: Mar 1, 2025
• Deposit Date: Nov 27, 2024
• Publicly Available Date: Jan 7, 2025
• Journal: Software Quality Journal
• Print ISSN: 0963-9314
• Electronic ISSN: 1573-1367
• Publisher: Springer Verlag
• Peer Reviewed: Peer Reviewed
• Volume: 33
• Issue: 1
• Article Number: 7
• DOI: https://doi.org/10.1007/s11219-024-09704-1
• Keywords: Domain-specific analysis, Error-type metrics, Software quality assurance, Risk categorisation, Software fault prediction, Principal component analysis, Extreme gradient boosting model
• Public URL: https://uwe-repository.worktribe.com/output/13468915
• Publisher URL: https://link.springer.com/journal/11219
• Additional Information: This is the version accepted for publication