The happy state of property-based testing in C#

07 Jul 2024

I recently spotted Stevan's very thorough blog post about the sad state of property-based testing libraries on Hacker News. I emailed to make him aware of CsCheck which has both stateful and parallel testing, and an innovative approach to shrinking. In reply he encouraged me to write something about these improvements, hopefully to do some bridge building between the communities, as there are too many silos of information.

Random shrinking and other innovations

Most property-based testing libraries are a port of QuickCheck and share its tree-based shrinking. This is where a lazy tree of possible simpler samples is composed and tested. CsCheck differs in that generation and shrinking are both based on random samples. Each generated sample comes with a Size proxy which can be used to compare for shrinking. Only samples with a smaller Size are tested during shrinking.

This gives the following advantages over tree based shrinking libraries:

• Automatic shrinking. Gen classes are composable with no need for Arb classes. So less boilerplate.
• Random testing and shrinking are parallelized. This and PCG make it very fast.
• Shrunk cases have a seed value. Simpler examples can easily be reproduced.
• Shrinking can be continued later to give simpler cases for high dimensional problems.
• Parallel testing and random shrinking work well together. Repeat is not needed.

A more detailed comparison can be found here.

Other innovative functionality in CsCheck:

• Check.SampleModelBased - Stateful model-based testing. This is the simplest and most powerful form of property-based testing.
• Check.SampleMetamorphic - Metamorphic testing. This is two path testing. Useful when a model can't be found that wouldn't just be a reimplementation.
• Check.SampleConcurrent - Parallel random testing. Robust concurrency testing. A perfect match for random shrinking.
• Check.Faster - Statistical performance testing. Correct statistical testing over a range of inputs. BenckmarkDotNet still doesn't have this.
• Check.Hash - Regression testing without the need for committing data files while also giving detailed information of any change.
• Causal.Profile - Causal profiling (idea from Emery Berger). Find the regions of code that are the bottleneck.

C# community pitch

It's not so easy in the C# community to get blog posts or new OSS libraries noticed. There doesn't seem to be anywhere you can submit them. I don't think there has been much of a take up of property-based testing in the community outside of some people who have used F# before.

A good example of the kind of limitation this creates is the ability to design a simple allocation algorithm of an integer over a list of weights. The algorithm seems like it should be easy (it's not), but everywhere I've seen it done it's not been correct, from multiple finance companies, twitter, excel and stackoverflow. The only correct algorithms I've seen coded are mine (1,2,3) found by using CsCheck.

Other blog examples of CsCheck use in optimising numeric algorithms and writing a high performance SIEVE LRU cache.

Some places to use random testing:

• Serialization - the number of bugs seen in serialization code is almost criminal given how easy it is to roundtrip test serialization using random testing.
• Caches and collections - often a key part of server and client-side code these can be tested against a suitable simplified test model with Model-Based testing.
• Calculations and algorithms - often possible to generalize examples for calculations and algorithms and check the result given the input. Algorithm often have properties they must guarantee. Rounding error issues automatically tested.
• Code refactoring - keep a copy of the original code with the test, refactor for simplicity and performance, safe in the knowledge it still produces the same results. Pair with a Faster test to monitor the relative performance over a range of inputs. Or if a copy is not feasible create a Regression test to comprehensively make sure there is no change.
• Multithreading and concurrency - test on the same object instance across multiple threads and examples. Shrink even works for Concurrency testing.

Why use it is discussed further here.