Generating Fixed Test Suites with QuickCheck

Posted by on Oct 16, 2014 in Blog

Generating Fixed Test Suites with QuickCheck The great strength of QuickCheck is that we never run out of tests to run: QuickCheck can always find more and more combinations of things to test, and so rare bugs that depend on several features interacting can eventually be found. Nevertheless, sometimes you just want to test fast, and be sure that you did indeed test everything at least once. QuickCheck is able to generate and run saved test suites with a variety of properties, which can be used as a fast sanity check in between longer test runs using random generation of test cases. In this blog post, we’ll show how to use recent additions to QuickCheck to generate better test suites. The example we’re going to use is a simple specification of the Erlang process registry, which does both positive and negative testing, and moreover kills processes randomly to test the registry’s behaviour when processes crash. You can find the source code we’re starting from here. It’s a state machine specification, with a property instrumented to collect the proportions of each command in the generated tests. prop_registry() -> ?FORALL(Cmds, commands(?MODULE), begin [catch unregister(N) || N <- ?names], {H, S, Res} = run_commands(?MODULE,Cmds), pretty_commands(?MODULE, Cmds, {H, S, Res}, aggregate(command_names(Cmds), Res ==...

Learn More

Getting Better Statistics from State Machine Tests

Posted by on Oct 15, 2014 in Blog

Getting Better Statistics from State Machine Tests One of the risks of property based testing is that, if your tests pass, then you never actually get to see your test data. You could potentially be running a large number of trivial tests, and ending up with a false sense of security as a result. To counter this, it’s important to collect statistics from your tests, and check that the distribution of tests is reasonable. This blog post shows you how to use one of the new features of QuickCheck state machines to get better statistics than you could before. A Simple Example As our example, we’re going to use a QuickCheck model of the Erlang process registry, which you can find here. This model combines positive and negative testing of register, unregister, and whereis, and moreover kills processes randomly to test the registry’s behaviour in the presence of crashes. Here is the property that tests the model: prop_registry() -> eqc:numtests(1000, eqc_statem:show_states( ?FORALL(Cmds, commands(?MODULE), begin [catch unregister(N) || N <- ?names], {H, S, Res} = run_commands(?MODULE,Cmds), pretty_commands(?MODULE, Cmds, {H, S, Res}, aggregate(command_names(Cmds), ?IMPLIES(Res/=precondition_failed, Res == ok))) end))). It specifies that we should run 1,000 tests, and that the model states should be pretty-printed when a test fails. It generates test...

Learn More