Shrinking and Equality Testing with FsCheck

In last week’s Property Testing with FsCheck, we saw how to write a couple of simple tests to check that an implementation of .Equals() was correct. We didn’t see what happens when the test fails, nor what we can do to make that failure easier to understand.

When comparing two Tag instances for equality, I want to treat tags that differ only in letter case as equal - if someone tags one document as “important” and another document as “Important”, they both have the same tag.

Let’s add a pair of tests that verify this behavior - one that checks lowercasing isn’t significant, the other to check that uppercasing is also ignored.

[Property]
public void Equals_ComparingWithLowercasedTag_ReturnsTrue(Tag origin)
{
    var other = new Tag(origin.Label.ToLowerInvariant());
    origin.Equals(other).Should().BeTrue();
}

[Property]
public void Equals_ComparingWithUppercasedTag_ReturnsTrue(Tag origin)
{
    var other = new Tag(origin.Label.ToUpperInvariant());
    origin.Equals(other).Should().BeTrue();
}

In both tests, I rely on FsCheck to create a tag for testing. Then, I transmogrify that tag and compare it with the original.

Running the tests, we get some test failures - here’s one example:

FsCheck.Xunit.PropertyFailedException

Falsifiable, after 1 test (0 shrinks) (StdGen (387719983,296318424)):
Original:
[cr11jEjE9ocycycw2h7r7rZ7waUkUkS0pdNdNd]

Remember that FsCheck generates random test cases, so you won’t get exactly the same failure case if you’re trying this at home.

While this failure is useful - and in this simple case, enough for me to identify the underlying cause of the bug - it would be nice if the sample failure was simpler.

Fortunately, FsCheck supports this with the idea of Shrinkers. A Shrinker takes an existing object and creates a simpler, smaller variation that will then be retested - allowing FsCheck to reduce the result to the simplest possible failure.

We can shrink a Tag by removing one character from the label. Keeping in mind our requirement that a tag label starts with a letter, I wrote this shrinker:

public static IEnumerable<Tag> Shrink(Tag tag)
{
    var chars = tag.Label.ToCharArray();
    if (chars.Length <= 1)
    {
        yield break;
    }

    for (int index = 0; index < chars.Length; index++)
    {
        var label = new string(
            chars.Where((c, i) => i != index).ToArray());
        if (Tag.IsValidLabel(label))
        {
            yield return new Tag(label);
        }
    }
}

Notice that the shrinker returns IEnumerable<Tag> - it generates many different tags, each smaller than the original. If the tag label is just one character, we can’t shrink any further. Otherwise, we try deleting each character, in turn, returning new tags for any sequence that is still a valid tag label.

We need to modify the definition of our Arbitrary<Tag> function to incorporate the shrinker:

public static Arbitrary<Tag> Tags() 
    => Arb.From(TagGenerators.Generate(), TagGenerators.Shrink);

Now our test failure is a bit clearer:

Falsifiable, after 1 test (16 shrinks) (StdGen (870949865,296318443)):
Original:
[Hytv1v1SSdvBvjpxt]
Shrunk:
[B]

Making the obvious change to our implementation of Tag.Equals(Tag), we find our tests now pass:

Ok, passed 100 tests.