Slides available as pdf.
The code is available on Github <3
- Mutation Testing for Java: PiTest
- Mutation Testing for JS, C# and Scala: Stryker
- Consumer Contracts
Mutation Testing could take a long time. When is a good time to run it?
It is absolutly true that mutation testing can take a long time, so it’s not always practical to run it in every build. Often, I see teams run them at 2 moments:
- An individual developer uses Mutation Testing whenever they want to validate if the tests they have written are actually valuable;
- In the nightly build we run Mutation Testing as a reference point. The build won’t fail if a certain % of mutation coverage isn’t reached. Every so often, we go through the report just to validate if there isn’t a big discrepancy between Line Coverage and Mutation Coverage.
In short: Use mutation testing as a tool to support you in writing better tests, not to punish you for not getting a certain %.
You showed how we could inject random failures with ToxiProxy. Wouldn’t that make the build fail at random? (Non-deterministic tests)
Yes… the use of “jitter” (to randomly include latency) might indeed make your build fail, which is why I wouldn’t recommend keeping those random tests active.
Where ToxiProxy shines, which I didn’t mention enough during my presentation, are some of the other “toxics”. These inject issues but they aren’t random!
- Setting a fixed amount of latency (to test timeouts);
- Setting a fixed amount of bandwidth (to test slow connections);
- Setting a data limit. Once the limit is reached, the connection is cut;
Since they aren’t random in nature, these should be just fine to keep in your active testset.