Testing Techniques

Google I/O 2014

Andrew Gerrand

Video

This talk was presented at golang-syd in July 2014.

Watch the talk on YouTube

Testing Go code

Go has a built-in testing framework.

It is provided by the testing package and the go test command.

Here is a complete test file that tests the strings.Index function:

package strings_test

import (
    "strings"
    "testing"
)

func TestIndex(t *testing.T) {
    const s, sep, want = "chicken", "ken", 4
    got := strings.Index(s, sep)
    if got != want {
        t.Errorf("Index(%q,%q) = %v; want %v", s, sep, got, want)
    }
}

Table-driven tests

Go's struct literal syntax makes it easy to write table-driven tests:

func TestIndex(t *testing.T) {
    var tests = []struct {
        s   string
        sep string
        out int
    }{
        {"", "", 0},
        {"", "a", -1},
        {"fo", "foo", -1},
        {"foo", "foo", 0},
        {"oofofoofooo", "f", 2},
        // etc
    }
    for _, test := range tests {
        actual := strings.Index(test.s, test.sep)
        if actual != test.out {
            t.Errorf("Index(%q,%q) = %v; want %v", test.s, test.sep, actual, test.out)
        }
    }
}

T

The *testing.T argument is used for error reporting:

t.Errorf("got bar = %v, want %v", got, want)
t.Fatalf("Frobnicate(%v) returned error: %v", arg, err)
t.Logf("iteration %v", i)

And enabling parallel tests:

t.Parallel()

And controlling whether a test runs at all:

if runtime.GOARCH == "arm" {
    t.Skip("this doesn't work on ARM")
}

Running tests

The go test command runs tests for the specified package.
(It defaults to the package in the current directory.)

$ go test
PASS

$ go test -v
=== RUN TestIndex
--- PASS: TestIndex (0.00 seconds)
PASS

To run the tests for all my projects:

$ go test github.com/nf/...

Or for the standard library:

$ go test std

Test coverage

The go tool can report test coverage statistics.

$ go test -cover
PASS
coverage: 96.4% of statements
ok      strings    0.692s

The go tool can generate coverage profiles that may be interpreted by the cover tool.

$ go test -coverprofile=cover.out
$ go tool cover -func=cover.out
strings/reader.go:    Len             66.7%
strings/strings.go:   TrimSuffix     100.0%
... many lines omitted ...
strings/strings.go:   Replace        100.0%
strings/strings.go:   EqualFold      100.0%
total:                (statements)    96.4%

Coverage visualization

$ go tool cover -html=cover.out

An example program

outyet is a web server that announces whether or not a particular Go version has been tagged.

go get github.com/golang/example/outyet

Testing HTTP clients and servers

The net/http/httptest package provides helpers for testing code that makes or serves HTTP requests.

httptest.Server

An httptest.Server listens on a system-chosen port on the local loopback interface, for use in end-to-end HTTP tests.

type Server struct {
    URL      string // base URL of form http://ipaddr:port with no trailing slash
    Listener net.Listener

    // TLS is the optional TLS configuration, populated with a new config
    // after TLS is started. If set on an unstarted server before StartTLS
    // is called, existing fields are copied into the new config.
    TLS *tls.Config

    // Config may be changed after calling NewUnstartedServer and
    // before Start or StartTLS.
    Config *http.Server
}

func NewServer(handler http.Handler) *Server

func (*Server) Close() error

httptest.Server in action

This code sets up a temporary HTTP server that serves a simple "Hello" response.

// +build ignore,OMIT

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"net/http/httptest"
)

func main() {

    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()

    res, err := http.Get(ts.URL)
    if err != nil {
        log.Fatal(err)
    }

    greeting, err := ioutil.ReadAll(res.Body)
    res.Body.Close()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("%s", greeting)

httptest.ResponseRecorder

httptest.ResponseRecorder is an implementation of http.ResponseWriter that records its mutations for later inspection in tests.

type ResponseRecorder struct {
    Code      int           // the HTTP response code from WriteHeader
    HeaderMap http.Header   // the HTTP response headers
    Body      *bytes.Buffer // if non-nil, the bytes.Buffer to append written data to
    Flushed   bool
}

httptest.ResponseRecorder in action

By passing a ResponseRecorder into an HTTP handler we can inspect the generated response.

// +build ignore,OMIT

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
)

func main() {

    handler := func(w http.ResponseWriter, r *http.Request) {
        http.Error(w, "something failed", http.StatusInternalServerError)
    }

    req, err := http.NewRequest("GET", "http://example.com/foo", nil)
    if err != nil {
        log.Fatal(err)
    }

    w := httptest.NewRecorder()
    handler(w, req)

    fmt.Printf("%d - %s", w.Code, w.Body.String())

Race Detection

A data race occurs when two goroutines access the same variable concurrently and at least one of the accesses is a write.

To help diagnose such bugs, Go includes a built-in data race detector.

Pass the -race flag to the go tool to enable the race detector:

$ go test -race mypkg    // to test the package
$ go run -race mysrc.go  // to run the source file
$ go build -race mycmd   // to build the command
$ go install -race mypkg // to install the package

Testing with concurrency

When testing concurrent code, there's a temptation to use sleep;
it's easy and works most of the time.

But "most of the time" isn't always and flaky tests result.

We can use Go's concurrency primitives to make flaky sleep-driven tests reliable.

Finding errors with static analysis: vet

The vet tool checks code for common programmer mistakes:

bad printf formats,
bad build tags,
bad range loop variable use in closures,
useless assignments,
unreachable code,
bad use of mutexes,
and more.

Usage:

go vet [package]

Testing from the inside

Most tests are compiled as part of the package under test.

This means they can access unexported details, as we have already seen.

Testing from the outside

Occasionally you want to run your tests from outside the package under test.

(Test files as package foo_test instead of package foo.)

This can break dependency cycles. For example:

The testing package uses fmt.
The fmt tests must import testing.
So the fmt tests are in package fmt_test and can import both testing and fmt.

Mocks and fakes

Go eschews mocks and fakes in favor of writing code that takes broad interfaces.

For example, if you're writing a file format parser, don't write a function like this:

func Parse(f *os.File) error

instead, write functions that take the interface you need:

func Parse(r io.Reader) error

(An *os.File implements io.Reader, as does bytes.Buffer or strings.Reader.)

Subprocess tests

Sometimes you need to test the behavior of a process, not just a function.

func Crasher() {
    fmt.Println("Going down in flames!")
    os.Exit(1)
}

To test this code, we invoke the test binary itself as a subprocess:

func TestCrasher(t *testing.T) {
    if os.Getenv("BE_CRASHER") == "1" {
        Crasher()
        return
    }
    cmd := exec.Command(os.Args[0], "-test.run=TestCrasher")
    cmd.Env = append(os.Environ(), "BE_CRASHER=1")
    err := cmd.Run()
    if e, ok := err.(*exec.ExitError); ok && !e.Success() {
        return
    }
    t.Fatalf("process ran with err %v, want exit status 1", err)
}

Thank you