Property-Based Testing With FsCheck

Introduction

This article introduces the concept of property-based testing which is another approach to verify that a function provides an expected result. The complete source code of the project can be accessed on GitHub.

False safety of traditional unit-testing approach

Let's imagine that we have an array of musicians where we define a musician by his/her name and an overall number of people who have listened to that musician:

type Artist = {
        name: string
        listeners: int
    }

And also, we have a function that orders a descendant array of type above by listeners count.

let orderArtistsByListenersCount artists =
        let ordered =
                artists
                    |> Array.sortBy (fun i -> -i.listeners)
        Success(ordered)

We could write a unit-test to verify that this function works correctly with xUnit just as below.

[<Fact>]
    let orderArtistsByListenersCount_returns_expected_result() =
        let Satie = {name = "Erik Satie"; listeners = 750000}
        let Chopin = {name ="Frederic Chopin"; listeners = 1200000}
        let Barber = {name = "Samuel Barber"; listeners = 371000}
        let artists = [|Satie; Chopin; Barber|]
        let result = orderArtistsByListenersCount artists
        match result with
        | Success s ->
            Assert.Equal(s.[0], Chopin)
            Assert.Equal(s.[1], Satie)
            Assert.Equal(s.[2], Barber)
        | Failure _ -> Assert.True(false)

Looks like we're fine now but in fact, nothing stops another developer from implementing a function which just satisfies the magic numbers provided in test.

let orderArtistsByListenersCount artists =
        let Satie = {name = "Erik Satie"; listeners = 750000}
        let Chopin = {name ="Frederic Chopin"; listeners = 1200000}
        let Barber = {name = "Samuel Barber"; listeners = 371000}
        let artists = [|Chopin; Satie; Barber|]
        Success(artists)

Actually, nothing verifies sortedness of an array.

Property-based testing with FsCheck

On the contrary, property-based testing verifies whether our result satisfies the properties that we define. For example, the below code checks whether each item of the sorted array has bigger or same amount of listeners as the next item

open FsCheck.Xunit

let ``pairs from collection should be ordered`` orderFn artists =
        let orderedArtists = artists |> orderFn
        match orderedArtists with
        | Success s -> s |> Array.pairwise |> Array.forall (fun (x,y) -> x.listeners >= y.listeners)
        | Failure _ -> false

[<Property>]
let pairwise x =
        ``pairs from collection should be ordered`` orderArtistsByListenersCount x

What FsCheck does is that it generates several hundred random inputs to test that the property is satisfied upon all possible input values.

Next example checks whether the result array contains same items as the input array.

let ``should be permutation of original elements`` orderFn artists =
        let orderedArtists = artists |> orderFn
        match orderedArtists with
        | Success s -> s |> List.ofArray |> isPermutationOf (List.ofArray artists)
        | Failure _ -> false

[<Property>]
let isPermutation x =
        ``should be permutation of original elements`` orderArtistsByListenersCount x

 Implementation of isPermutation can be found here.

Conclusion

Examples of the code above verify not just that the function complies with some magic-number inputs but examine fundamental properties of the result, which adds extra security to your tests.