Application of Mutation testing in Safety- Critical Embedded Systems: A Case Study

Abstract

Mutation testing is a technique used for evaluating test efficiencies, by analyzing whether existing tests could detect minor modifications inserted in the source code. Despite its proven benefits and added value to the verification and validation process, mutation testing is yet to become a widespread practice in safety-critical software development, mostly due to issues around its scalability in industrial environments. In this case study, conducted at Knorr-Bremse Rail Systems, we created a lightweight mutation testing tool, tailored to the specific test environment of the company, showing how such tools can be created with a notably smaller workload, than estimated in previously published case studies. Mutation testing was used to analyze automatically generated and manually complemented coverage-based tests of an entire braking system component. Mutation testing was able to reveal deficiencies not uncovered by standardized, coverage-based testing. The experience added to the body of knowledge on the application of mutation testing, in safety-critical embedded systems, strengthening the fault-finding capability of mutation testing reported by earlier related studies, but pointing out how the one-mutant- per-line optimization was less useful in the given setting. The findings resulted in the definitive, strategic implementation of the created tool within the company’s component testing workflow and could help replicate the results in other case studies, aiding companies in introducing mutation testing in their work environment.