Property Testing with FsCheck

I’ve been writing some property tests with FsCheck and to say that the process has been bittersweet is a bit of an understatement. The good news is that writing property tests has successfully uncovered bugs. The bad news is that writing property tests has successfully uncovered bugs.

I have a personal project makes extensive use of semantic types. These are simple types that provide a strongly typed encapsulation of key concepts, allowing the compiler to enforce correct parameter passing.

One of these semantic types is Tag - a kind of label that can be attached to other things in the problem domain as a way of adding additional information about that item. The Tag implementation looks something like this:

public sealed class Tag
{
    public string Label { get; }

    public Tag(string label)
    {
        Label = label;
    }

    public override bool Equals(object obj)
    {
        var other = obj as Tag;
        if (other == null)
        {
            return false;
        }

        return string.Equals(
            Label, other.Label, StringComparison.Ordinal);
    }

    public override int GetHashCode() => Label.GetHashCode();

    public override string ToString() => "[" + Label + "]";
}

I’ve simplified this class a little to allow us to concentrate on its essential characteristics - the full implementation includes complete documentation as well as a number of operators.

Property testing allows me to declare the characteristics (or properties) that the implementation should possess, with the framework taking care of test generation.

For example, two of the properties of the Equals method can be expressed with these tests:

[Property]
public void Equals_GivenSelf_ReturnsTrue(Tag tag)
{
    tag.Equals(tag).Should().BeTrue();
}

[Property]
public void Equals_GivenDifferentTag_ReturnsFalse(Tag tag, Tag other)
{
    tag.Equals(other).Should().BeFalse();
}

Interestingly, while I haven’t been able to find any documentation that guarantees the two tags will be different, the test hasn’t ever failed for that reason.

To make these tests work, we need to teach FsCheck how to generate random instances of Tag for the tests to use. Here’s how we do that.

A valid tag is made from an identifier like string, starting with a letter and composed of letters and digits. We start by creating a pair of generators, one to generate a sequence of letters, and the other to generate a sequence of letters and digits.

public static class CharGenerators 
{
    public static Gen<char> Letters()
        => from c in Arb.Default.Char().Generator
            where Char.IsLetter(c)
            select c;

    public static Gen<char> LettersAndDigits()
        => from c in Arb.Default.Char().Generator
            where Char.IsLetterOrDigit(c)
            select c;
}

You can see how the FsCheck library meets the conventions required for the use of LINQ syntax, allowing generators to be defined using a very declarative syntax.

With the above generators available, I can create a new generator that generates valid tag identifiers:

public static Gen<string> Identifiers(int maxLength) => 
    from length in Gen.Choose(0, maxLength)
    from head in CharGenerators.Letters()
    from tail in Gen.ArrayOf(length - 1, CharGenerators.LettersAndDigits())
    select head + new string(tail);

This now allows me to write a simple generator that creates Tags:

public static Gen<Tag> Tags() => 
    from name in StringGenerators.Identifiers(50)
    select new Tag(name);

This pattern of simpler generators being used to define more complex generators seems to repeat a lot when working with FsCheck. We then need to tell FsCheck where to start when it needs a Tag:

public class CoreArbitrary
{
    public static Arbitrary<Tag> Tags() => 
        Arb.From(TagGenerators.Tags());
}

Finally, we put an attribute on our test class to hook everything up:

[Properties(Arbitrary = new[] { typeof(CoreArbitrary) })]
public class TagTests
{
    // ... elided ...
}

When activated, FsCheck randomly generates 100 different tags and puts the code through it’s paces, checking that our implementation has the desired characteristics.

Updated 26/6: Fixed some bugs in the code.