Testing

"Elm Test" is the combination of the Elm test library and the elm-test Node.js command-line application. In addition to unit testing, Elm Test is equipped with support for fuzzing, namely, the use of randomness to help generate large numbers of relevant test cases.

See this and this for info to install elm-test. You'll need Node.js and the Node package manager (npm).

% npm install -g elm-test

This kind of command is often run with "superuser" privileges to install things globally, for all user accounts:

% sudo npm install -g elm-test

But if you still get some EACCES errors, try the suggestion here to change npm's default directory — thanks Justin!

Then, once elm-test is installed, create an Elm project directory and get going:

% elm init
% elm-test init  # `elm-test install elm-explorations/test` and create tests/Example.elm
% elm-test       # Run all exposed Test values in *.elm files in tests/

A Bug

Here's a buggy function:

module Main exposing (..)

absoluteValue n =
  if n == -99 then n
  else if n > 0 then n
  else -n

The Test module provides an API for defining tests, which declare Expectations (defined in Expect) about the results of computations. Here's a simple

test : String -> (() -> Expectation) -> Test

in the auto-generated tests/Example.elm file, using the built-in Basics.abs function as a reference solution:

module Example exposing (..)

import Test exposing (Test)
import Expect exposing (Expectation)
import Fuzz exposing (Fuzzer)
import Main


suite : Test
suite =
  Test.describe "Test suite for Main module"
    [ Test.test "absoluteValue -5" <|
        \() ->
          let input = -5 in
          input
            |> Main.absoluteValue
            |> Expect.equal (Basics.abs input)
    ]

A function with a type of the form () -> T is called a thunk: it is a delayed expression of type T that will evaluate only when the thunk is forced (i.e. called with the unit argument). We'll consider these concepts in detail when we study lazy evaluation.

Then run:

% elm-test

...

TEST RUN PASSED

Duration: 147 ms
Passed:   3
Failed:   0

Test.fuzz and Fuzz.int can be used to generate (100) random tests:

    ...
    , Test.fuzz Fuzz.int "absoluteValue (?? : Int)" <|
        \n ->
          n
            |> Main.absoluteValue
            |> Expect.equal (Basics.abs n)
    ...

Probably won't catch the bug:

% elm-test

...

TEST RUN PASSED

...

(Did you notice the reproducable randomness with seeds in the output?)

Run more random tests with an option in the Elm code...

...
, Test.fuzzWith { runs = 500 } Fuzz.int "absoluteValue (?? : Int)" <|
...

... or with a command-line option to the JavaScript test runner:

% elm-test --fuzz=500

Probably still won't catch the bug: "It's possible for Fuzz.int to generate any 32-bit integer, signed or unsigned, but it favors numbers between -50 and 50 and especially zero." Use Fuzz.intRange instead to sample from a different range:

    ...
    , Test.fuzz (Fuzz.intRange -100 100) "absoluteValue (?? : [-100,100])" <|
        \n ->
          n
            |> Main.absoluteValue
            |> Expect.equal (Basics.abs n)
    ]

Maybe now you'll catch the bug:

% elm-test

elm-test 0.19.0-rev6
--------------------

Running 3 tests. To reproduce these results, run: elm-test --fuzz 100 --seed 346611013530184
XXX/tests/Example.elm

↓ Example
↓ Test suite for Main module
✗ absoluteValue (?? : [-100,100])

Given -99

    -99
    ╷
    │ Expect.equal
    ╵
    99



TEST RUN FAILED

Duration: 161 ms
Passed:   2
Failed:   1

ArrayHeap

After we implemented ArrayHeap.elm, we wrote a very simple ArrayHeapTest.elm module module to test whether our ArrayHeap-backed implementation of simpleHeapSort agreed with List.sort... on a single test case. Now let's add some fuzzing. Take a look at how testSuite is implemented. And then run:

% elm-test ArrayHeapTest.elm

Note that we are testing only the public API, in particular, the insert and deleteMin. If we wanted to define tests that check invariants about the internal complete binary tree representation, then we would have to define some tests from within the module (or expose more internal type and value definitions during development and testing).

Try injecting a bug into ArrayHeap and see what happens.


"Reading"