Gophers With Hammers

Josh Bleecher Snyder
PayPal
josharian@gmail.com
@offbymany


* Go was designed with tools in mind. (Rob Pike)

* Designed with tools in mind

Simple, regular syntax
Simple semantics
Batteries included

* Tools everywhere

- go
- gofmt, goimports
- godoc
- go test [-cover] [-race]
- go vet
- gofix, gofmt -r, eg
- oracle
- golint
- godep

and more...

* go command

	$ go list -f '{{.Deps}}' bytes
	[errors io runtime sync sync/atomic unicode unicode/utf8 unsafe]

* gofmt

from

	for{
	fmt.Println(      "I feel pretty." );
	       }

to

	for {
		fmt.Println("I feel pretty.")
	}

* godoc

	$ godoc strings Repeat
	func Repeat(s string, count int) string
	    Repeat returns a new string consisting of count copies of the string s.

* go vet

Oops

	if suffix != ".md" || suffix != ".markdown" {

Flagged

	suspect or: suffix != ".md" || suffix != ".markdown"

* go tool cover -mode=set

	func Repeat(s string, count int) string {
		b := make([]byte, len(s)*count)
		bp := 0
		for i := 0; i < count; i++ {
			bp += copy(b[bp:], s)
		}
		return string(b)
	}

to

	func Repeat(s string, count int) string {
		GoCover.Count[0] = 1
		b := make([]byte, len(s)*count)
		bp := 0
		for i := 0; i < count; i++ {
			GoCover.Count[2] = 1
			bp += copy(b[bp:], s)
		}
		GoCover.Count[1] = 1
		return string(b)
	}

* go test -cover

	$ go test -coverprofile=c.out strings
	ok  	strings	0.455s	coverage: 96.9% of statements

	$ go tool cover -func=c.out
	strings/reader.go:	Len				66.7%
	strings/reader.go:	Read				100.0%
	strings/reader.go:	ReadAt				100.0%
	strings/reader.go:	ReadByte			100.0%
	strings/reader.go:	UnreadByte			100.0%
	strings/reader.go:	ReadRune			100.0%
	strings/reader.go:	UnreadRune			100.0%
	strings/reader.go:	Seek				90.9%
	strings/reader.go:	WriteTo				83.3%
	...

	$ go tool cover -html=c.out
	# opens a browser window, shows line-by-line coverage


* Tools to make tools

- text/template
- go/build
- go/doc
- go/format
- go/{parser,token,ast,printer}
- go.tools/go/types and friends

and more...

* Hammers are fun!

# Why to write your own tools: Fun, learning, profit

* impl

Generate implementation stubs given an interface.

	go get github.com/josharian/impl

Generate

	$ impl 'f *File' io.Reader
	func (f *File) Read(p []byte) (n int, err error) {
	}

from

	package io

	type Reader interface {
		Read(p []byte) (n int, err error)
	}

* impl

Generate

	$ impl 'f *File' io.ReadWriter
	func (f *File) Read(p []byte) (n int, err error) {
	}

	func (f *File) Write(p []byte) (n int, err error) {
	}

from

	package io

	type ReadWriter interface {
		Reader
		Writer
	}

* impl

Generate

	$ impl 'c *Ctx' http.Handler
	func (c *Ctx) ServeHTTP(http.ResponseWriter, *http.Request) {
	}

from

	package http

	type Handler interface {
		ServeHTTP(ResponseWriter, *Request)
	}

* Plan

*Find*import*path*and*interface*name*

	http.Handler ⇒ net/http, Handler

Parse interface

	net/http, Handler ⇒ {{"ServeHTTP", {{"", "http.ResponseWriter"}, {"", "*http.Request"}}, {}}}}

Generate output

	{{"ServeHTTP", {{"", "http.ResponseWriter"}, {"", "*http.Request"}}, {}}}} ⇒ profit!

* goimports ftw

	import "golang.org/x/tools/imports"

.play hammers/importpath.go /func main/,/^}/

* Hello, AST

    *ast.File {
    .  Package: 1:1
    .  Name: *ast.Ident {
    .  .  NamePos: 1:9
    .  .  Name: "hack"
    .  }
    .  Decls: []ast.Decl (len = 2) {
    .  .  0: *ast.GenDecl {
    .  .  .  TokPos: 1:15
    .  .  .  Tok: import
    .  .  .  Lparen: -
    .  .  .  Specs: []ast.Spec (len = 1) {
    .  .  .  .  0: *ast.ImportSpec {
    .  .  .  .  .  Path: *ast.BasicLit {
    .  .  .  .  .  .  ValuePos: 1:22
    .  .  .  .  .  .  Kind: STRING
    .  .  .  .  .  .  Value: "\"net/http\""
    .  .  .  .  .  }

[truncated]


* Extract the import path

	import (
		"go/parser"
		"go/token"
	)

.play hammers/extractpath.go /func main/,/^}/

* Extract the interface name

	import "go/ast"

.play hammers/extractiface.go /func main/,/^}/

A `GenDecl` can have many `Specs`

	var (
		r io.Reader
		w io.Writer
	)

* Plan

Find import path and interface name

	http.Handler ⇒ net/http, Handler

*Parse*interface*

	net/http, Handler ⇒ {{"ServeHTTP", {{"", "http.ResponseWriter"}, {"", "*http.Request"}}, {}}}}

Generate output

	{{"ServeHTTP", {{"", "http.ResponseWriter"}, {"", "*http.Request"}}, {}}}} ⇒ profit!

* Data structures

Represent

	Read(p []byte) (n int, err error)

as

	Func{
		Name:   "Read",
		Params: []Param{{Name: "p", Type: "[]byte"}},
		Res: []Param{
			{Name: "n", Type: "int"},
			{Name: "err", Type: "error"},
		},
	},

* Data structures

.code hammers/types.go /type Func/,/^}/
.code hammers/types.go /type Param/,/^}/

* Find the code

	import "go/build"

.play hammers/findthecode.go /func main/,/^}/

* Find the interface declaration

	import "go/printer"

.play hammers/findtheifacedecl.go /func main/,/^}/

* Extract function names

No name? It's an embedded interface. Recurse.

	type ByteScanner interface {
	    ByteReader
	    UnreadByte() error
	}

* Extract params and results

No name? Just use `""`.

	type ByteWriter interface {
	    WriteByte(c byte) error
	}

* Qualify types

Types can be arbitrarily complicated.

	type CrazyGopher interface {
		CrazyGoph(int) func(chan<- [32]byte, map[string]int64) ([]rune, error)
	}

And we need to rewrite some of them.

	int ⇒ int
	*Request ⇒ *http.Request
	io.Reader ⇒ io.Reader
	func(io.Reader, chan map[S][]*T) ⇒ func(io.Reader, chan map[foo.S][]*foo.T))

* Qualify types

.play hammers/fulltype.go /func main/,/end main/

* Plan

Find import path and interface name

	http.Handler ⇒ net/http, Handler

Parse interface

	net/http, Handler ⇒ {{"ServeHTTP", {{"", "http.ResponseWriter"}, {"", "*http.Request"}}, {}}}}

*Generate*output*

	{{"ServeHTTP", {{"", "http.ResponseWriter"}, {"", "*http.Request"}}, {}}}} ⇒ profit!

* Method type

.code hammers/types.go /type Method/,/^}/
.code hammers/types.go /type Func/,/^}/
.code hammers/types.go /type Param/,/^}/

* Use text/template

.play hammers/codegen.go /func main/,/^}/

# Don't generate an AST. It's a lot of work, and Go is its own DSL.

* Ugly is ok

	import "go/format"

.play hammers/format.go /func main/,/^}/

* Great success

Full code plus tests at `github.com/josharian/impl`

* Tips

Use `go`get`-d` to download lots of code from `godoc.org/-/index`. (Don't forget to set a temporary `GOPATH`!)

Use (and improve) `github.com/yuroyoro/goast-viewer`.

You don't have to generate all the code. And generating data is even better.

The `go/ast` docs are your friend.

`go.tools/go/types` is powerful.

`go`generate` is coming.


* Nails!

- Break up long strings
- Enums and flags to Stringers
- Dynamic code analysis
- Vet checks
- Reflect ⇒ codegen
- Convention-based http dispatch
- Detect "last line" copy/paste bugs
- AST-aware diff, merge, blame; automated fork analysis
- Machine learning models of ASTs: anomaly detection, bug-prone code detection
- Code fingerprinting
- Examine usage patterns
- Compiler stress tests