RUB » Bibliotheksportal

Markus Dreher, Christopher Stefan Friedrich Huth

• Coverage-guided fuzzing, is a test technique, where the interfaces of a Program under Test (PUT) are fed with semi-random inputs, mutated to maximize the PUT’s coverage depending on dynamically collected feedback. When measuring tests, there is \(\textit {test effectiveness}\) and \(\textit {test efficiency}\). Test effectiveness can be measured in the number of bugs found during testing. Fuzzing has already been shown to be an effective method to detect bugs. Test efficiency, on the other hand, can be measured with the used resources to get a certain test effect. Achieving complete coverage during testing, in a simple case line coverage, is an immense amount of test creation effort, when only unit tests are used - and most of those unit tests do not add to a thorough overall test. Due to continuously growing software systems, a \(\textit {reasonable effort}\) should be put into creating manual unit test cases. In this paper we propose a combination of unit tests and fuzz tests to reduce the overall effort in test creation. While unit tests are indispensable for functional testing, fuzz tests shine to unveil unwanted non-functional behavior, i.e. generating unexpected edge cases. This results in an overall reduction in manual effort for test creation, while the test effectiveness remains equal, i.e. coverage. We introduce an effort estimation framework for a combined unit test and fuzz test strategy. Additionally, we conduct an empirical case study of multiple automotive network components indicating test creation effort savings of around 26%, when manual (non-functional) unit tests are partly automated with fuzz tests - all while fuzzing discovered the same bugs as tortuous unit tests.