Modern Testing Patterns


As presented @ Jfokus

Session Content

Slides

Slides available as pdf.

Video

Code

The code is available on Github <3

Discussed Topics:

Q&A

Mutation Testing could take a long time. When is a good time to run it?

It is absolutly true that mutation testing can take a long time, so it’s not always practical to run it in every build. Often, I see teams run them at 2 moments:

  • An individual developer uses Mutation Testing whenever they want to validate if the tests they have written are actually valuable;
  • In the nightly build we run Mutation Testing as a reference point. The build won’t fail if a certain % of mutation coverage isn’t reached. Every so often, we go through the report just to validate if there isn’t a big discrepancy between Line Coverage and Mutation Coverage.

In short: Use mutation testing as a tool to support you in writing better tests, not to punish you for not getting a certain %.

You showed how we could inject random failures with ToxiProxy. Wouldn’t that make the build fail at random? (Non-deterministic tests)

Yes… the use of “jitter” (to randomly include latency) might indeed make your build fail, which is why I wouldn’t recommend keeping those random tests active.

Where ToxiProxy shines, which I didn’t mention enough during my presentation, are some of the other “toxics”. These inject issues but they aren’t random!

  • Setting a fixed amount of latency (to test timeouts);
  • Setting a fixed amount of bandwidth (to test slow connections);
  • Setting a data limit. Once the limit is reached, the connection is cut;

Since they aren’t random in nature, these should be just fine to keep in your active testset.