Stupid Gopher Tricks
GolangUK
21 August 2015
Andrew Gerrand
Andrew Gerrand
A video of this talk was recorded at GolangUK in London.
2This talk is about things you might not know about Go.
Some of this stuff you can add to your Go vocabulary.
Other things are probably best left for special occasions.
3Here's a familiar type declaration:
type Foo struct { i int s string }
The latter part of a type declaration is the type literal:
struct { i int s string }
Other examples of type literals are int
and []string
,
which can also be declared as named types:
type Bar int type Qux []string
While we commonly use int
and []string
in variable declarations:
var i int var s []string
It is less common (but equally valid) to do the same with structs:
var t struct { i int s string }
An unnamed struct literal is often called an anonymous struct.
6A common use is providing data to templates:
// +build ignore
package main
import (
"log"
"os"
"strings"
"text/template"
)
func main() {
tmpl := template.Must(template.New("").Parse(strings.TrimSpace(`
Dear {{.Title}} {{.Lastname}},
Congratulations on reaching Level {{.Rank}}!
I'm sure your parents would say "Great job, {{.Firstname}}!"
Sincerely,
Rear Admiral Gopher
`)))
data := struct { Title string Firstname, Lastname string Rank int }{ "Dr", "Carl", "Sagan", 7, } if err := tmpl.Execute(os.Stdout, data); err != nil { log.Fatal(err) }
}
You could also use map[string]interface{}
,
but then you sacrifice performance and type safety.
The same technique can be used for encoding JSON objects:
// +build ignore
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
b, err := json.Marshal(struct { ID int Name string }{42, "The answer"}) if err != nil { log.Fatal(err) } fmt.Printf("%s\n", b)
}
And also decoding:
// +build ignore
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
var data struct { ID int Name string } err := json.Unmarshal([]byte(`{"ID": 42, "Name": "The answer"}`), &data) if err != nil { log.Fatal(err) } fmt.Println(data.ID, data.Name)
}
Structs can be nested to describe more complex JSON objects:
// +build ignore
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
var data struct { ID int Person struct { Name string Job string } } const s = `{"ID":42,"Person":{"Name":"George Costanza","Job":"Architect"}}` err := json.Unmarshal([]byte(s), &data) if err != nil { log.Fatal(err) } fmt.Println(data.ID, data.Person.Name, data.Person.Job)
}
In repeated literals like slices and maps, Go lets you omit the inner type name:
type Foo struct { i int s string } var s = []Foo{ {6 * 9, "Question"}, {42, "Answer"}, } var m = map[int]Foo{ 7: {6 * 9, "Question"}, 3: {42, "Answer"}, }
Combined with anonymous structs, this convenience shortens the code dramatically:
var s = []struct { i int s string }{ struct { i int s string }{6 * 9, "Question"}, struct { i int s string }{42, "Answer"}, } var t = []struct { i int s string }{ {6 * 9, "Question"}, {42, "Answer"}, }
These properties enable a nice way to express test cases:
func TestIndex(t *testing.T) { var tests = []struct { s string sep string out int }{ {"", "", 0}, {"", "a", -1}, {"fo", "foo", -1}, {"foo", "foo", 0}, {"oofofoofooo", "f", 2}, // etc } for _, test := range tests { actual := strings.Index(test.s, test.sep) if actual != test.out { t.Errorf("Index(%q,%q) = %v; want %v", test.s, test.sep, actual, test.out) } } }
You can go a step further and put the composite literal in the range statement itself:
func TestIndex(t *testing.T) { for _, test := range []struct { s string sep string out int }{ {"", "", 0}, {"", "a", -1}, {"fo", "foo", -1}, {"foo", "foo", 0}, {"oofofoofooo", "f", 2}, // etc } { actual := strings.Index(test.s, test.sep) if actual != test.out { t.Errorf("Index(%q,%q) = %v; want %v", test.s, test.sep, actual, test.out) } } }
But this is harder to read.
13
A struct field that has no name is an embedded field.
The embedded type's methods (and fields, if it is a struct)
are accessible as if they are part of the embedding struct.
// +build ignore
package main
import "fmt"
type A struct { s string } func (a A) String() string { return fmt.Sprintf("A's String method called: %v", a.s) } type B struct { A } func main() { b := B{} b.s = "some value" fmt.Println(b) }
Of course, you can embed fields in an anonymous struct.
It's common to protect a global variable with a mutex variable:
var ( viewCount int64 viewCountMu sync.Mutex )
By embedding a mutex in an anonymous struct, we can group the related values:
var viewCount struct { sync.Mutex n int64 }
Users of viewCount
access it like this:
viewCount.Lock() viewCount.n++ viewCount.Unlock()
And you can embed interfaces, too.
Here's a real example from Camlistore:
The function is expected to return a ReadSeekCloser
,
but the programmer only had a string.
Anonymous struct (and its standard library friends) to the rescue!
return struct { io.ReadSeeker io.Closer }{ io.NewSectionReader(strings.NewReader(s), 0, int64(len(s))), ioutil.NopCloser(nil), }
Interfaces can be anonymous, the most common being interface{}
.
But the interface needn't be empty:
// +build ignore
package main
import (
"bytes"
"fmt"
)
func main() {
var s interface { String() string } = bytes.NewBufferString("I'm secretly a fmt.Stringer!") fmt.Println(s.String())
}
Useful for a sly type assertion (from src/os/exec/exec_test.go
):
// Check that we can access methods of the underlying os.File. if _, ok := stdin.(interface { Fd() uintptr }); !ok { t.Error("can't access methods of underlying *os.File") }
A "method value" is what you get when you evaluate a method as an expression.
The result is a function value.
Evaluating a method from a type yields a function:
// +build ignore
package main
import (
"bytes"
"os"
)
func main() {
var f func(*bytes.Buffer, string) (int, error) var buf bytes.Buffer f = (*bytes.Buffer).WriteString f(&buf, "y u no buf.WriteString?") buf.WriteTo(os.Stdout)
}
Evaluating a method from a value creates a closure that holds that value:
// +build ignore
package main
import (
"bytes"
"os"
)
func main() {
var f func(string) (int, error) var buf bytes.Buffer f = buf.WriteString f("Hey... ") f("this *is* cute.") buf.WriteTo(os.Stdout)
}
The sync.Once
type is used to perform a task once with concurrency safety.
// Once is an object that will perform exactly one action. type Once struct { /* Has unexported fields. */ } func (o *Once) Do(f func())
This LazyPrimes
type computes a slice of prime numbers the first time it is used:
type LazyPrimes struct { once sync.Once // Guards the primes slice. primes []int } func (p *LazyPrimes) init() { // Populate p.primes with prime numbers. } func (p *LazyPrimes) Primes() []int { p.once.Do(p.init) return p.primes }
You can use method values to implement multiple HTTP handlers with one type:
type Server struct { // Server state. } func (s *Server) index(w http.ResponseWriter, r *http.Request) { /* Implementation. */ } func (s *Server) edit(w http.ResponseWriter, r *http.Request) { /* Implementation. */ } func (s *Server) delete(w http.ResponseWriter, r *http.Request) { /* Implementation. */ } func (s *Server) Register(mux *http.ServeMux) { mux.HandleFunc("/", s.index) mux.HandleFunc("/edit/", s.edit) mux.HandleFunc("/delete/", s.delete) }
In package os/exec
, the Cmd
type implements methods to set up
standard input, output, and error:
func (c *Cmd) stdin() (f *os.File, err error) func (c *Cmd) stdout() (f *os.File, err error) func (c *Cmd) stderr() (f *os.File, err error)
The caller handles each in the same way,
so it iterates over a slice of method values:
type F func(*Cmd) (*os.File, error) for _, setupFd := range []F{(*Cmd).stdin, (*Cmd).stdout, (*Cmd).stderr} { fd, err := setupFd(c) if err != nil { c.closeDescriptors(c.closeAfterStart) c.closeDescriptors(c.closeAfterWait) return err } c.childFiles = append(c.childFiles, fd) }
The Go spec defines a set of types as "comparable";
they may be compared with == and !=.
Bools, ints, floats, complex numbers, strings, pointers,
channels, structs, and interfaces are comparable.
// +build ignore
package main
import "fmt"
func main() {
var a, b int = 42, 42 fmt.Println(a == b) var i, j interface{} = a, b fmt.Println(i == j) var s, t struct{ i interface{} } s.i, t.i = a, b fmt.Println(s == t)
}
A struct is comparable only if its fields are comparable:
// +build ignore
package main
import "fmt"
func main() {
var q, r struct{ s []string } fmt.Println(q == r)
}
Any comparable type may be used as a map key.
a := map[int]bool{} a[42] = true type T struct { i int s string } b := map[*T]bool{} b[&T{}] = true c := map[T]bool{} c[T{37, "hello!"}] = true d := map[interface{}]bool{} d[42] = true d[&T{}] = true d[T{123, "four five six"}] = true d[ioutil.Discard] = true
An example from the Go continuous build infrastructure:
type builderRev struct { builder, rev string } var br = builderRev{"linux-amd64", "0cd299"}
We track in-flight builds in a map.
The pre-Go 1 way was to flatten the data to a string first:
inflight := map[string]bool{} inflight[br.builder + "-" + br.rev] = true
But with struct keys, you can avoid the allocation and have cleaner code:
inflight := map[builderRev]bool{} inflight[br] = true
An example of interface map keys from Docker's broadcastwriter
package:
type BroadcastWriter struct { sync.Mutex writers map[io.WriteCloser]struct{} } func (w *BroadcastWriter) AddWriter(writer io.WriteCloser) { w.Lock() w.writers[writer] = struct{}{} w.Unlock() } func (w *BroadcastWriter) Write(p []byte) (n int, err error) { w.Lock() for sw := range w.writers { if n, err := sw.Write(p); err != nil || n != len(p) { delete(w.writers, sw) } } w.Unlock() return len(p), nil }
A (very) contrived example: (Don't do this! Ever!)
// +build ignore
package main
import "fmt"
type cons struct { car string cdr interface{} } func (c cons) String() string { if c.cdr == nil || c.cdr == (cons{}) { return c.car } return fmt.Sprintf("%v %v", c.car, c.cdr) } func main() { m := map[cons]string{} c := cons{} for _, s := range []string{"life?", "with my", "I doing", "What am"} { c = cons{s, c} } m[c] = "No idea." fmt.Println(c, m[c]) }
For a simple counter, you can use the sync/atomic
package's
functions to make atomic updates without the lock.
func AddInt64(addr *int64, delta int64) (new int64) func CompareAndSwapInt64(addr *int64, old, new int64) (swapped bool) func LoadInt64(addr *int64) (val int64) func StoreInt64(addr *int64, val int64) func SwapInt64(addr *int64, new int64) (old int64)
First, define the global variable (appropriately documented!):
// viewCount must be updated atomically. var viewCount int64
Then increment it with AddInt64
:
count := atomic.AddInt64(&viewCount, 1)
The set are available for Int32
, Uint32
, Int64
, Uint64
, Pointer
, and Uintptr
.
Another option for sharing state is atomic.Value
.
For instance, to share configuration between many goroutines:
type Config struct { Timeout time.Duration } var config atomic.Value
To set or update, use the Store
method:
config.Store(&Config{Timeout: 2*time.Second})
To read, each goroutine calls the Load
method:
cfg := config.Load().(*Config)
Note that storing different types in the same Value
will cause a panic.
The atomic.Value
primitive is the size of a single interface value:
package atomic type Value struct { v interface{} }
An interface value is represented by the runtime as two pointers:
one for the type, and one for the value.
// ifaceWords is interface{} internal representation. type ifaceWords struct { typ unsafe.Pointer data unsafe.Pointer }
To load and store an interface value atomically, it operates on the parts of the interface value with atomic.LoadPointer
and atomic.StorePointer
.
The Store
method first validates the input:
func (v *Value) Store(x interface{}) { if x == nil { panic("sync/atomic: store of nil value into Value") }
Then uses unsafe
to cast the current and new interface{}
values to ifaceWords
:
// ... vp := (*ifaceWords)(unsafe.Pointer(v)) xp := (*ifaceWords)(unsafe.Pointer(&x))
(This allows us to get at the internals of those interface values.)
31Spin while loading the type field:
// ... for { typ := LoadPointer(&vp.typ)
If it's nil
the this is the first time the value has been stored.
Put a sentinel value (max uintptr) in the type field to "lock" it while we work with it:
// ... if typ == nil { if !CompareAndSwapPointer(&vp.typ, nil, unsafe.Pointer(^uintptr(0))) { continue // Someone beat us to it. Wait. }
Store the data field, then the type field, and we're done!
// ... StorePointer(&vp.data, xp.data) StorePointer(&vp.typ, xp.typ) return }
If this isn't the first store, check whether a store is already happening:
// ... if uintptr(typ) == ^uintptr(0) { continue // First store in progress. Wait. }
Sanity check whether the type changed:
// ... if typ != xp.typ { panic("sync/atomic: store of inconsistently typed value into Value") }
If the type field is what we expect, go ahead and atomically store the value:
// ... StorePointer(&vp.data, xp.data) return } }
The Load
method first loads the interface's type field:
func (v *Value) Load() (x interface{}) { vp := (*ifaceWords)(unsafe.Pointer(v)) typ := LoadPointer(&vp.typ)
Then, check whether a store has happened, or is happening:
// ... if typ == nil || uintptr(typ) == ^uintptr(0) { return nil }
Otherwise, load the data field and return both type and data as a new interface value:
// ... data := LoadPointer(&vp.data) xp := (*ifaceWords)(unsafe.Pointer(&x)) xp.typ = typ xp.data = data return }
Usually you don't want or need the stuff in this package.
Try channels or the sync
package first.
You almost certainly shouldn't write code like the atomic.Value
implementation.
(And I didn't show all of it; the real code has hooks into the runtime.)
Sometimes you need to test the behavior of a process, not just a function.
func Crasher() { fmt.Println("Going down in flames!") os.Exit(1) }
To test this code, we invoke the test binary itself as a subprocess:
func TestCrasher(t *testing.T) { if os.Getenv("BE_CRASHER") == "1" { Crasher() return } cmd := exec.Command(os.Args[0], "-test.run=TestCrasher") cmd.Env = append(os.Environ(), "BE_CRASHER=1") err := cmd.Run() if e, ok := err.(*exec.ExitError); ok && !e.Success() { return } t.Fatalf("process ran with err %q, want exit status 1", err) }
Go's CPU and memory profilers report data for an entire process.
To profile just one side of a concurrent operation, you can use a sub-process.
This benchmark from the net/http
package spawns a child process to make
requests to a server running in the main process.
func BenchmarkServer(b *testing.B) { b.ReportAllocs() // Child process mode; if url := os.Getenv("TEST_BENCH_SERVER_URL"); url != "" { n, err := strconv.Atoi(os.Getenv("TEST_BENCH_CLIENT_N")) if err != nil { panic(err) } for i := 0; i < n; i++ { res, err := Get(url) // ... } os.Exit(0) return } // ...
// ... res := []byte("Hello world.\n") ts := httptest.NewServer(HandlerFunc(func(rw ResponseWriter, r *Request) { rw.Header().Set("Content-Type", "text/html; charset=utf-8") rw.Write(res) })) defer ts.Close() cmd := exec.Command(os.Args[0], "-test.run=XXXX", "-test.bench=BenchmarkServer$") cmd.Env = append([]string{ fmt.Sprintf("TEST_BENCH_CLIENT_N=%d", b.N), fmt.Sprintf("TEST_BENCH_SERVER_URL=%s", ts.URL), }, os.Environ()...) if out, err := cmd.CombinedOutput(); err != nil { b.Errorf("Test failure: %v, with output: %s", err, out) } }
To run:
$ go test -run=XX -bench=BenchmarkServer -benchtime=15s -cpuprofile=http.prof $ go tool pprof http.test http.prof
$ go help list usage: go list [-e] [-f format] [-json] [build flags] [packages] List lists the packages named by the import paths, one per line.
Show the packages under a path:
$ go list golang.org/x/oauth2/... golang.org/x/oauth2 ... golang.org/x/oauth2/vk
Show the standard library:
$ go list std archive/tar ... unsafe
Show all packages:
$ go list all
The -json
flag tells you everything the go tool knows about a package:
$ go list -json bytes { "Dir": "/Users/adg/go/src/bytes", "ImportPath": "bytes", "Name": "bytes", "Doc": "Package bytes implements functions for the manipulation of byte slices.", "Target": "/Users/adg/go/pkg/darwin_amd64/bytes.a", "Goroot": true, "Standard": true, "Stale": true, "Root": "/Users/adg/go", "GoFiles": [ "buffer.go", "bytes.go", "bytes_decl.go", "reader.go" ], ... }
It's easy to write programs that consume this data.
43
The go tool's documented Package
struct describes all the possible fields:
type Package struct { Dir string // directory containing package sources ImportPath string // import path of package in dir ImportComment string // path in import comment on package statement Name string // package name Doc string // package documentation string Target string // install path Shlib string // the shared library that contains this package (only set when -linkshared) Goroot bool // is this package in the Go root? Standard bool // is this package part of the standard Go library? Stale bool // would 'go install' do anything for this package? Root string // Go root or Go path dir containing this package // (more fields on next slide) }
type Package struct { // (more fields on previous slide) // Source files GoFiles []string // .go source files (excluding CgoFiles, TestGoFiles, XTestGoFiles) CgoFiles []string // .go sources files that import "C" IgnoredGoFiles []string // .go sources ignored due to build constraints CFiles []string // .c source files CXXFiles []string // .cc, .cxx and .cpp source files MFiles []string // .m source files HFiles []string // .h, .hh, .hpp and .hxx source files SFiles []string // .s source files SwigFiles []string // .swig files SwigCXXFiles []string // .swigcxx files SysoFiles []string // .syso object files to add to archive // (more fields on next slide) }
type Package struct { // (more fields on previous slide) // Cgo directives CgoCFLAGS []string // cgo: flags for C compiler CgoCPPFLAGS []string // cgo: flags for C preprocessor CgoCXXFLAGS []string // cgo: flags for C++ compiler CgoLDFLAGS []string // cgo: flags for linker CgoPkgConfig []string // cgo: pkg-config names // Dependency information Imports []string // import paths used by this package Deps []string // all (recursively) imported dependencies // Error information Incomplete bool // this package or a dependency has an error Error *PackageError // error loading package DepsErrors []*PackageError // errors loading dependencies TestGoFiles []string // _test.go files in package TestImports []string // imports from TestGoFiles XTestGoFiles []string // _test.go files outside package XTestImports []string // imports from XTestGoFiles }
That's a ton of information, so what can we do with it?
The -f
flag lets you use Go's text/template
package to format the ouput.
Show package doc strings:
$ go list -f '{{.Doc}}' golang.org/x/oauth2/... Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests. Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0". ...
Show the Go files in a package:
$ go list -f '{{.GoFiles}}' bytes [buffer.go bytes.go bytes_decl.go reader.go]
Make the output cleaner with a join
:
$ go list -f '{{join .GoFiles " "}}' bytes buffer.go bytes.go bytes_decl.go reader.go
With template logic we can test packages for certain conditions.
Find standard libraries that lack documentation:
$ go list -f '{{if not .Doc}}{{.ImportPath}}{{end}}' std internal/format internal/trace
Find packages that depend (directly or indirectly) on a given package:
$ go list -f '{{range .Deps}}{{if eq . "golang.org/x/oauth2"}}{{$.ImportPath}}{{end}}{{end}}' all golang.org/x/build/auth golang.org/x/build/buildlet golang.org/x/build/cmd/buildlet ...
Find packages that are broken somehow (note -e
):
$ go list -e -f '{{with .Error}}{{.}}{{end}}' all package github.com/golang/oauth2: code in directory /Users/adg/src/github.com/golang/oauth2 expects import "golang.org/x/oauth2" ...
Things get interesting once we start using the shell.
For instance, we can use go
list
output as input to the go tool.
Test all packages except vendored packages:
$ go test $(go list ./... | grep -v '/vendor/')
Test the dependencies of a specific package:
$ go test $(go list -f '{{join .Deps " "}}' golang.org/x/oauth2)
The same, but don't test the standard library:
$ go test $( go list -f '{{if (not .Goroot)}}{{.ImportPath}}{{end}}' $( go list -f '{{join .Deps " "}}' golang.org/x/oauth2 ) )
for pkg in $(go list golang.org/x/oauth2/...); do wc -l $(go list -f '{{range .GoFiles}}{{$.Dir}}/{{.}} {{end}}' $pkg) | \ tail -1 | awk '{ print $1 " '$pkg'" }' done | sort -nr
The output:
617 golang.org/x/oauth2/google 600 golang.org/x/oauth2 357 golang.org/x/oauth2/internal 160 golang.org/x/oauth2/jws 147 golang.org/x/oauth2/jwt 112 golang.org/x/oauth2/clientcredentials 22 golang.org/x/oauth2/paypal 16 golang.org/x/oauth2/vk 16 golang.org/x/oauth2/odnoklassniki 16 golang.org/x/oauth2/linkedin 16 golang.org/x/oauth2/github 16 golang.org/x/oauth2/facebook
( echo "digraph G {" go list -f '{{range .Imports}}{{printf "\t%q -> %q;\n" $.ImportPath .}}{{end}}' \ $(go list -f '{{join .Deps " "}}' time) time echo "}" ) | dot -Tsvg -o time-deps.svg
There are many more fun corners of Go.
Can you find them all? :-)
Read the docs, explore, and have fun!
52Andrew Gerrand