Writing Concurrent Tests In Golang

Golang provides excellent concurrency mechanism, including some built-in functionalities to run tests faster. In this blog post we’ll explore some useful techniques in running concurrent tests in Go. However, we will not cover basic knowledge of Golang syntax, concepts and how to write tests in Golang in this post.

Basic tests

Imagine we want to implement some helper functions for the slice data type.

package slices

// Unique returns true if slice does not contain any duplicates.
func Unique[T comparable](slice []T) bool {
	exists := make(map[T]struct{})

	for _, elem := range slice {
		if _, ok := exists[elem]; ok {
			return false
		}

		exists[elem] = struct{}{}
	}

	return true
}

// Exists returns true if elem is found in slice.
func Exists[T comparable](slice []T, elem T) bool {
	for _, e := range slice {
		if e == elem {
			return true
		}
	}

	return false
}

We write some simple test cases to verify our helper functions as below.

func TestUnique(t *testing.T) {
	got := Unique([]string{"foo", "bar"})
	assert.True(t, got)
}

func TestExists(t *testing.T) {
	got := Exists([]string{"foo", "bar"}, "bar")
	assert.True(t, got)
}

Executing a test, or a suite of tests, can be achieved by the following command.

go clean -testcache && go test -v ./...

The output we got may look like this.

=== RUN   TestUnique
--- PASS: TestUnique (0.00s)
=== RUN   TestExists
--- PASS: TestExists (0.00s)
PASS
ok      github.com/xamenyap/go-concurrent-tests-demo/slices     0.440s

As can be seen in the log, all tests run sequentially, and it took them 0.440s to complete. Depending on your machine, the time to complete the test may differ, but the key point here is that Golang reports how long our tests take to run.

Testing with Concurrency

We can make our tests concurrent by adding t.Parallel().

func TestUnique(t *testing.T) {
	t.Parallel()
	got := Unique([]string{"foo", "bar"})
	assert.True(t, got)
}

func TestExists(t *testing.T) {
	t.Parallel()
	got := Exists([]string{"foo", "bar"}, "bar")
	assert.True(t, got)
}

It must be noted that even though the method name is Parallel(), it’s not guaranteed that the tests will run in parallel, because concurrency is not parallelism.

Running the test again yields a different log, and we can also notice that it executed: 0.097s versus 0.440s. Again the numbers here may differ from machine to machine.

=== RUN   TestUnique
=== PAUSE TestUnique
=== RUN   TestExists
=== PAUSE TestExists
=== CONT  TestUnique
--- PASS: TestUnique (0.00s)
=== CONT  TestExists
--- PASS: TestExists (0.00s)
PASS
ok      github.com/xamenyap/go-concurrent-tests-demo/slices     0.097s

If we look at the log closely we see that the tests paused twice. Why? In fact whenever t.Parallel() is called the test where it occupies will pause, and continue in a concurrent manner. To have a better understanding of how Golang orchestrates the concurrent tests, we may add another test without t.Parallel() which will produce some interesting results. Let’s add the Index helper function to examine this idea.

package slices
// Index returns the index of the first occurrence of elem in slice.
// If elem is not found, then -1 is returned.
func Index[T comparable](slice []T, elem T) int {
	for i, e := range slice {
		if e == elem {
			return i
		}
	}

	return -1
}

The test for Index can be written as below.

func TestIndex(t *testing.T) {
	idx := Index([]string{"foo", "bar"}, "bar")
	assert.Equal(t, 1, idx)
}

Running the test again yields this output.

=== RUN   TestUnique
=== PAUSE TestUnique
=== RUN   TestExists
=== PAUSE TestExists
=== RUN   TestIndex
--- PASS: TestIndex (0.00s)
=== CONT  TestUnique
--- PASS: TestUnique (0.00s)
=== CONT  TestExists
--- PASS: TestExists (0.00s)
PASS
ok      github.com/xamenyap/go-concurrent-tests-demo/slices     0.316s

This is interesting because even though TestIndex is the last test to run, it’s actually the first to finish. Looking at the log carefully, we see that TestUnique and TestExists paused, and then resumed only after TestIndex had finished. Therefore, it can be implied that that whenever t.Parallel() is called, the tests will pause, and Golang will execute them later after all sequential tests.

Concurrency in sub tests

What about sub tests? We may try to update TestExists to examine another data type other than string.

func TestExists(t *testing.T) {
	t.Parallel()

	t.Run("strings", func(t *testing.T) {
		got := Exists([]string{"foo", "bar"}, "bar")
		assert.True(t, got)
	})

	t.Run("integers", func(t *testing.T) {
		got := Exists([]int{1, 2, 3}, 2)
		assert.True(t, got)
	})
}

The output will look like this

=== RUN   TestUnique
=== PAUSE TestUnique
=== RUN   TestExists
=== PAUSE TestExists
=== RUN   TestIndex
--- PASS: TestIndex (0.00s)
=== CONT  TestUnique
--- PASS: TestUnique (0.00s)
=== CONT  TestExists
=== RUN   TestExists/strings
=== RUN   TestExists/integers
--- PASS: TestExists (0.00s)
    --- PASS: TestExists/strings (0.00s)
    --- PASS: TestExists/integers (0.00s)
PASS
ok      github.com/xamenyap/go-concurrent-tests-demo/slices     0.471s

From the log we can see that TestExists ran concurrently with TestUnique. However, the sub tests within TestExists were executed sequentially. To maximize the power of concurrency, we actually have to call t.Parallel() within each sub test of TestExists.

func TestExists(t *testing.T) {
	t.Parallel()

	t.Run("strings", func(t *testing.T) {
		t.Parallel()
		got := Exists([]string{"foo", "bar"}, "bar")
		assert.True(t, got)
	})

	t.Run("integers", func(t *testing.T) {
		t.Parallel()
		got := Exists([]int{1, 2, 3}, 2)
		assert.True(t, got)
	})
}

Running the tests again we will get the following output.

=== RUN   TestUnique
=== PAUSE TestUnique
=== RUN   TestExists
=== PAUSE TestExists
=== RUN   TestIndex
--- PASS: TestIndex (0.00s)
=== CONT  TestUnique
--- PASS: TestUnique (0.00s)
=== CONT  TestExists
=== RUN   TestExists/strings
=== PAUSE TestExists/strings
=== RUN   TestExists/integers
=== PAUSE TestExists/integers
=== CONT  TestExists/strings
=== CONT  TestExists/integers
--- PASS: TestExists (0.00s)
    --- PASS: TestExists/strings (0.00s)
    --- PASS: TestExists/integers (0.00s)
PASS
ok      github.com/xamenyap/go-concurrent-tests-demo/slices     0.431s

It’s clear that the sub tests for strings and integers paused and then resumed later to run concurrently with each other during the life cycle of the tests. Therefore, to fully benefit from concurrency in tests, we have to call t.Parallel() not just in the test itself, but also in the sub tests within the test. And if the sub tests contain any further sub tests, then this rule applies recursively. This is an important point, because it cannot be inferred Parallel()’s document at the moment this blog post is written.

Parallel signals that this test is to be run in parallel with (and only with) other parallel tests. When a test is run multiple times due to use of -test.count or -test.cpu, multiple instances of a single test never run in parallel with each other.

Digging deeper into Golang doc, we learn that the number of concurrent tests that are executed this way is controlled by the -parallel flag.

Concurrent tests across multiple packages

One must note that tests in the same package run sequentially by default, unless t.Parallel() is called as explained in the previous sections. On the contrary, tests in different packages will run concurrently by default, which can be controlled by the -p flag. Let’s check this by creating another package to provide some map helper.

package maps

// Keys returns all keys in m.
func Keys[K comparable, V any](m map[K]V) []K {
	ks := make([]K, 0)
	for k := range m {
		ks = append(ks, k)
	}

	return ks
}

The test for Keys can be written as below.

func TestKeys(t *testing.T) {
	ks := Keys(map[int]float32{
		1: 1.5,
		2: 3.5,
	})

	assert.ElementsMatch(t, ks, []int{1, 2})
}

If we invoke the test command again, tests for package slices and tests for package maps will run concurrently by default.

=== RUN   TestKeys
--- PASS: TestKeys (0.00s)
PASS
ok      github.com/xamenyap/go-concurrent-tests-demo/maps       0.399s
=== RUN   TestUnique
=== PAUSE TestUnique
=== RUN   TestExists
=== PAUSE TestExists
=== RUN   TestIndex
--- PASS: TestIndex (0.00s)
=== CONT  TestUnique
--- PASS: TestUnique (0.00s)
=== CONT  TestExists
=== RUN   TestExists/strings
=== PAUSE TestExists/strings
=== RUN   TestExists/integers
=== PAUSE TestExists/integers
=== CONT  TestExists/strings
=== CONT  TestExists/integers
--- PASS: TestExists (0.00s)
    --- PASS: TestExists/strings (0.00s)
    --- PASS: TestExists/integers (0.00s)
PASS
ok      github.com/xamenyap/go-concurrent-tests-demo/slices     0.099s

Cleanup in concurrent tests

Now that we know a set of tests, or sub tests calling t.Parallel() will pause, and only continue after all sequential tests finish. This means defer in a test (to close a database connection, or a file for example) must be used with caution: a defer statement within a sub test that runs in parallel with others will be executed when the top level test function returns, even before the supposedly concurrent sub tests resume. Therefore, it’s always recommended to use t.Cleanup() to handle post-processing hooks in our tests.

Summary

here are what we learned in this blog post:

Tests in the same package run sequentially by default. However, they can run concurrently by calling the t.Parallel() method. It must be noted that t.Parallel() must be called by the test, and all of its sub tests to fully utilize the power of concurrency. The number of tests that are executed concurrently this way is determined by the -parallel flag, whose default value is equal to GOMAXPROCS.
Tests in different packages are run concurrently by default. The number of packages that run concurrently is determined by the -p flag, with the default value being equal to the number of CPUs available.
Use t.Cleanup() to handle all clean up procedures in your tests instead of relying on defer.

All the source code for this demo can be accessed here

Basic tests#

Testing with Concurrency#

Concurrency in sub tests#

Concurrent tests across multiple packages#

Cleanup in concurrent tests#

Summary#