High Performance Apps with Go on App Engine

Google I/O, May 2013

Video

This talk was presented at Google I/O in May 2013.

Watch the talk on YouTube

Overview

• Why Go + App Engine
• History, Status
• Gopher Mart
• Finding performance bottlenecks
• Defer work
• Batching
• Caching
• Concurrency
• Control variance

Why Go + App Engine

• Go compiles to native code
• App Engine is an auto-scaling low-maintenance platform
• Fastest runtime on App Engine: starts fast, runs fast

History

• Go runtime for App Engine unveiled at Google I/O 2011
• Steady growth, several high-profile users

Turkey doodle (Nov 2011)

• Written by a Go newcomer, launched in under 24 hours
• Half the latency of a Python 2.7 version
• Launched on google.com front page

go.dev/s/turkey-doodle

Santa Tracker (Dec 2012)

• Nearly 5000 queries per second
• Did not make any children cry

Gopher Mart

Gopher Mart

• Imagine your app is the Gopher Mart, the one-stop-shop for all your gopher needs

Gopher Mart

• Each checkout gopher is an app instance
• Each shopping gopher is a user request
• The scheduler directs requests to instances
• A checkout gopher can deal with one customer at a time
• Checkout gophers (instances) can be hired/fired, but there's overhead

Gopher Mart

• Gophers waiting in queues get grumpy
• Let's make it fast

Finding performance bottlenecks

• Understand why your checkout gophers might work slowly
• Measure first

• Measure periodically; performance characteristics changes over time
• Go 1.1 brings lots of performance improvements

Appstats

• Appstats traces API RPCs, shows timeline
• github.com/mjibson/appstats

• Gopher Mart baseline:

Performance Techniques

Defer work

• Not all work needs to be done during the request
• Use appengine/taskqueue or appengine/delay to move non-critical work outside the request scope
• Gopher Mart can replace slow mail.Send with quick taskqueue.Add

Defer work II

Import "appengine/delay" and transform

    sendReceipt(c, user.Current(c).Email, b.String())

func sendReceipt(c appengine.Context, dst, body string) {

into

    sendReceipt.Call(c, user.Current(c).Email, b.String())

var sendReceipt = delay.Func("send-receipt", func(c appengine.Context, dst, body string) {

Batching

• Buying 10 boxes of Gopher Flakes is easier than buying a single box 10 times
• Calling for a price check on three items is faster than three separate price checks
• Use GetMulti instead of Get, PutMulti instead of Put, etc.

    var items []*Item
    for _, key := range keys {
        item := new(Item)
        if err := datastore.Get(c, key, item); err != nil {
            // ...
        }
        items = append(items, item)
    }

Batching II

    items := make([]Item, len(keys))
    if err := datastore.GetMulti(c, keys, items); err != nil {
        // ...
    }

Caching

• Checkout gophers can make notes and memorize things
• Using datastore is fine, but it can be slow
• Using memcache is good (shared among instances), but remember it can disappear
• Using local memory is okay too, but it can disappear as well

Concurrency

• A price check may delay dealing with a shopping gopher
• RPC-bound requests are very common, and often least-cacheable
• Get your checkout gopher doing something else while waiting

Concurrency II

    var lists []List
    var items []Item
    _, err := datastore.NewQuery("List").GetAll(c, &lists)
    if err != nil { /* ... */ }
    _, err := datastore.NewQuery("Item").GetAll(c, &items)
    if err != nil { /* ... */ }
    // write response

Concurrency III

• Run queries concurrently
• Easy way to speed up RPC-bound requests

    var lists []List
    var items []Item
    errc := make(chan error)
    go func() {
        _, err := datastore.NewQuery("List").GetAll(c, &lists)
        errc <- err
    }()
    go func() {
        _, err := datastore.NewQuery("Item").GetAll(c, &items)
        errc <- err
    }()
    err1, err2 := <-errc, <-errc
    if err1 != nil || err2 != nil { /* ... */ }
    // write response

• Visit Sameer Ajmani's talk (today, 4:25PM, Room 7)

Control variance

• Some shopping gophers can take much longer than others
• Variance in request processing is very common (e.g. 99% take 10ms, 1% take 100ms)
• Some requests are harder than others
• Infrastructure is reliable, but not perfect

Control variance II

• Effects of variance can cascade
• Variable requests make scheduling harder, requires more instances (billed)
• We want to control variance

"The Tail at Scale", Dean, Barroso; Commun. ACM 56, 2

Control variance III

• For example, storing in memcache is usually an optimisation, and thus optional
• Bound time spent on optional work

• First approach: save to memcache asynchronously

func myHandler(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)
    // ...
    // regular request handling
    // ...

    go memcache.Set(c, &memcache.Item{
        Key:   key,
        Value: data,
    })
}

• Problem: Response will be returned to user as soon as handler returns, and outstanding API calls will be canceled

Control variance IV

• Solution: Use Go's concurrency primitives to timeout waiting

func myHandler(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)
    // ...
    // regular request handling
    // ...

    // Save to memcache, but only wait up to 3ms.
    done := make(chan bool, 1) // NB: buffered
    go func() {
        memcache.Set(c, &memcache.Item{
            Key:   key,
            Value: data,
        })
        done <- true
    }()
    select {
    case <-done:
    case <-time.After(3 * time.Millisecond):
    }
}

Before and After

Baseline:

Defer work:

Batching:

Summary

• Finding performance bottlenecks
• Defer work
• Batching
• Caching
• Concurrency
• Control variance

Finally...

• go.dev
• developers.google.com/appengine/docs/go/
• go.dev/s/io13-ae-talk (this talk, plus gophermart app)
• github.com/mjibson/appstats

More Go things:

• Go in Production (office hours), 3:30PM-4:15PM, Cloud Sandbox
• Advanced Go Concurrency Patterns, 4:25PM, Room 7
• Fireside Chat with the Go Team, 5:20PM, Room 2
• Go App Engine (office hours), 1:45PM-2:30PM tomorrow, Cloud Sandbox

Thank you