Text file
talks/2012/splash.slide
1 Go at Google
2 SPLASH, Tucson, Oct 25, 2012
3
4 Rob Pike
5 Google, Inc.
6 https://go.dev
7
8 * Preamble
9
10 * What is Go?
11
12 Go is:
13
14 - open source
15 - concurrent
16 - garbage-collected
17 - efficient
18 - scalable
19 - simple
20 - fun
21 - boring (to some)
22
23 .link / go.dev
24
25 * History
26
27 Design began in late 2007.
28
29 Key players:
30
31 - Robert Griesemer, Rob Pike, Ken Thompson
32 - Later: Ian Lance Taylor, Russ Cox
33
34 Became open source in November 2009.
35
36 Developed entirely in the open; very active community.
37 Language stable as of Go 1, early 2012.
38
39 * Go at Google
40
41 Go is a programming language designed by Google to help solve Google's problems.
42
43 Google has big problems.
44
45 * Big hardware
46
47 .image splash/datacenter.jpg
48
49 * Big software
50
51 - C++ (mostly) for servers, plus lots of Java and Python
52 - thousands of engineers
53 - gazillions of lines of code
54 - distributed build system
55 - one tree
56
57 And of course:
58
59 - zillions of machines, which we treat as a modest number of compute clusters
60
61 Development at Google can be slow, often clumsy.
62
63 But it _is_ effective.
64
65 * The reason for Go
66
67 Goals:
68
69 - eliminate slowness
70 - eliminate clumsiness
71 - improve effectiveness
72 - maintain (even improve) scale
73
74 Go was designed by and for people who write—and read and debug and maintain—large software systems.
75
76 Go's purpose is _not_ research into programming language design.
77
78 Go's purpose is to make its designers' programming lives better.
79
80 * Today's theme
81
82 A talk about software engineering more than language design.
83
84 To be more accurate:
85
86 - Language design in the service of software engineering.
87
88 In short:
89
90 - How does a language help software engineering?
91
92 * Features?
93
94 Reaction upon launch:
95 My favorite feature isn't in Go! Go Sucks!
96
97 This misses the point.
98
99 * Pain
100
101 What makes large-scale development hard with C++ or Java (at least):
102
103 - slow builds
104 - uncontrolled dependencies
105 - each programmer using a different subset of the language
106 - poor program understanding (documentation, etc.)
107 - duplication of effort
108 - cost of updates
109 - version skew
110 - difficulty of automation (auto rewriters etc.): tooling
111 - cross-language builds
112
113 Language _features_ don't usually address these.
114
115 * Focus
116
117 In the design of Go, we tried to focus on solutions to _these_ problems.
118
119 Example: indentation for structure vs. C-like braces
120
121 * Dependencies in C and C++
122
123 * A personal history of dependencies in C
124
125 `#ifdef`
126
127 `#ifndef` "guards":
128
129 #ifndef _SYS_STAT_H_
130
131 1984: `<sys/stat.h>` times 37
132
133 ANSI C and `#ifndef` guards:
134
135 - dependencies accumulate
136 - throw includes at the program until it compiles
137 - no way to know what can be removed
138
139 * A personal history of dependencies in Plan 9's C
140
141 Plan 9, another take:
142
143 - no `#ifdefs` (or `#ifndefs`)
144 - documentation and topological sort
145 - easy to find out what can be removed
146
147 Need to document dependencies, but much faster compilation.
148
149 In short:
150
151 - _ANSI_C_made_a_costly_mistake_ in requiring `#ifndef` guards.
152
153 * A personal history of dependencies in C++
154
155 C++ exacerbated the problem:
156
157 - one `#include` file per class
158 - code (not just declarations) in `#include` files
159 - `#ifndef` guards persist
160
161 2004: Mike Burrows and Chubby: `<xxx>` times 37,000
162
163 1984: Tom Cargill and pi
164
165 * A personal history of dependencies at Google
166
167 Plan 9 demo: a story
168
169 Early Google: one `Makefile`
170
171 2003: `Makefile` generated from per-directory `BUILD` files
172
173 - explicit dependencies
174 - 40% smaller binaries
175
176 Dependencies still not checkable!
177
178 * Result
179
180 To build a large Google binary on a single computer is impractical.
181
182 In 2007, instrumented the build of a major Google binary:
183
184 - 2000 files
185 - 4.2 megabytes
186 - 8 gigabytes delivered to compiler
187 - 2000 bytes sent to compiler for every C++ source byte
188 - it's real work too: `<string>` for example
189 - hours to build
190
191 * Tools can help
192
193 New distributed build system:
194
195 - no more `Makefile` (still uses `BUILD` files)
196 - many buildbots
197 - much caching
198 - much complexity (a large program in its own right)
199
200 Even with Google's massive distributed build system, a large build still takes minutes.
201 (In 2007 that binary took 45 minutes; today, 27 minutes.)
202
203 Poor quality of life.
204
205 * Enter Go
206
207 While that build runs, we have time to think.
208
209 Want a language to improve the quality of life.
210
211 And dependencies are only one such problem....
212
213 * Primary considerations
214
215 Must work at scale:
216
217 - large programs
218 - large teams
219 - large number of dependencies
220
221 Must be familiar, roughly C-like
222
223 * Modernize
224
225 The old ways are _old_.
226
227 Go should be:
228
229 - suitable for multicore machines
230 - suitable for networked machines
231 - suitable for web stuff
232
233 * The design of Go
234
235 From a software engineering perspective.
236
237 * Dependencies in Go
238
239 * Dependencies
240
241 Dependencies are defined (syntactically) in the language.
242
243 Explicit, clear, computable.
244
245 import "encoding/json"
246
247 Unused dependencies cause error at compile time.
248
249 Efficient: dependencies traversed once per source file...
250
251 * Hoisting dependencies
252
253 Consider:
254 `A` imports `B` imports `C` but `A` does not directly import `C`.
255
256 The object code for `B` includes all the information about `C` needed to import `B`.
257 Therefore in `A` the line
258
259 import "B"
260
261 does not require the compiler to read `C` when compiling `A`.
262
263 Also, the object files are designed so the "export" information comes first; compiler doing import does not need to read whole file.
264
265 Exponentially less data read than with `#include` files.
266
267 With Go in Google, about 40X fanout (recall C++ was 2000x)
268 Plus in C++ it's general code that must be parsed; in Go it's just export data.
269
270 * No circular imports
271
272 Circular imports are illegal in Go.
273
274 The big picture in a nutshell:
275
276 - occasional minor pain,
277 - but great reduction in annoyance overall
278 - structural typing makes it less important than with type hierarchies
279 - keeps us honest!
280
281 Forces clear demarcation between packages.
282
283 Simplifies compilation, linking, initialization.
284
285 * API design
286
287 Through the design of the standard library, great effort spent on controlling dependencies.
288
289 It can be better to copy a little code than to pull in a big library for one function.
290 (A test in the system build complains if new core dependencies arise.)
291
292 Dependency hygiene trumps code reuse.
293
294 Example:
295 The (low-level) `net` package has own `itoa` to avoid dependency on the big formatted I/O package.
296
297 * Packages
298
299 * Packages
300
301 Every Go source file, e.g. `"encoding/json/json.go"`, starts
302
303 package json
304
305 where `json` is the "package name", an identifier.
306 Package names are usually concise.
307
308 To use a package, need to identify it by path:
309
310 import "encoding/json"
311
312 And then the package name is used to qualify items from package:
313
314 var dec = json.NewDecoder(reader)
315
316 Clarity: can always tell if name is local to package from its syntax: `Name` vs. `pkg.Name`.
317 (More on this later.)
318
319 Package combines properties of library, name space, and module.
320
321 * Package paths are unique, not package names
322
323 The path is `"encoding/json"` but the package name is `json`.
324 The path identifies the package and must be unique.
325 Project or company name at root of name space.
326
327 import "google/base/go/log"
328
329 Package name might not be unique; can be overridden. These are both `package`log`:
330
331 import "log" // Standard package
332 import googlelog "google/base/go/log" // Google-specific package
333
334 Every company might have its own `log` package; no need to make the package name unique.
335
336 Another: there are many `server` packages in Google's code base.
337
338 * Remote packages
339
340 Package path syntax works with remote repositories.
341 The import path is just a string.
342
343 Can be a file, can be a URL:
344
345 go get github.com/4ad/doozer // Command to fetch package
346
347 import "github.com/4ad/doozer" // Doozer client's import statement
348
349 var client doozer.Conn // Client's use of package
350
351 * Go's Syntax
352
353 * Syntax
354
355 Syntax is not important...
356
357 - unless you're programming
358 - or writing tools
359
360 Tooling is essential, so Go has a clean syntax.
361 Not super small, just clean:
362
363 - regular (mostly)
364 - only 25 keywords
365 - straightforward to parse (no type-specific context required)
366 - easy to predict, reason about
367
368 * Declarations
369
370 Uses Pascal/Modula-style syntax: name before type, more type keywords.
371
372 var fn func([]int) int
373 type T struct { a, b int }
374
375 not
376
377 int (*fn)(int[]);
378 struct T { int a, b; }
379
380 Easier to parse—no symbol table needed. Tools become easier to write.
381
382 One nice effect: can drop `var` and derive type of variable from expression:
383
384 var buf *bytes.Buffer = bytes.NewBuffer(x) // explicit
385 buf := bytes.NewBuffer(x) // derived
386
387 For more information:
388
389 .link /s/decl-syntax go.dev/s/decl-syntax
390
391 * Function syntax
392
393 Function on type `T`:
394
395 func Abs(t T) float64
396
397 Method of type `T`:
398
399 func (t T) Abs() float64
400
401 Variable (closure) of type `T`:
402
403 negAbs := func(t T) float64 { return -Abs(t) }
404
405 In Go, functions can return multiple values. Common case: `error`.
406
407 func ReadByte() (c byte, err error)
408
409 c, err := ReadByte()
410 if err != nil { ... }
411
412 More about errors later.
413
414 * No default arguments
415
416 Go does not support default function arguments.
417
418 Why not?
419
420 - too easy to throw in defaulted args to fix design problems
421 - encourages too many args
422 - too hard to understand the effect of the fn for different combinations of args
423
424 Extra verbosity may happen but that encourages extra thought about names.
425
426 Related: Go has easy-to-use, type-safe support for variadic functions.
427
428 * Naming
429
430 * Export syntax
431
432 Simple rule:
433
434 - upper case initial letter: `Name` is visible to clients of package
435 - otherwise: `name` (or `_Name`) is not visible to clients of package
436
437 Applies to variables, types, functions, methods, constants, fields....
438
439 That Is It.
440
441 Not an easy decision.
442 One of the most important things about the language.
443
444 Can see the visibility of an identifier without discovering the declaration.
445
446 Clarity.
447
448 * Scope
449
450 Go has very simple scope hierarchy:
451
452 - universe
453 - package
454 - file (for imports only)
455 - function
456 - block
457
458 * Locality of naming
459
460 Nuances:
461
462 - no implicit `this` in methods (receiver is explicit); always see `rcvr.Field`
463 - package qualifier always present for imported names
464 - (first component of) every name is always declared in current package
465
466 No surprises when importing:
467
468 - adding an exported name to my package cannot break your package!
469
470 Names do not leak across boundaries.
471
472 In C, C++, Java the name `y` could refer to anything
473 In Go, `y` (or even `Y`) is always defined within the package.
474 In Go, `x.Y` is clear: find `x` locally, `Y` belongs to it.
475
476 * Function and method lookup
477
478 Method lookup by name only, not type.
479 A type cannot have two methods with the same name, ever.
480 Easy to identify which function/method is referred to.
481 Simple implementation, simpler program, fewer surprises.
482
483 Given a method `x.M`, there's only ever one `M` associated with `x`.
484
485 * Semantics
486
487 * Basics
488
489 Generally C-like:
490
491 - statically typed
492 - procedural
493 - compiled
494 - pointers etc.
495
496 Should feel familiar to programmers from the C family.
497
498 * But...
499
500 Many small changes in the aid of robustness:
501
502 - no pointer arithmetic
503 - no implicit numeric conversions
504 - array bounds checking
505 - no type aliases
506 - `++` and `--` as statements not expressions
507 - assignment not an expression
508 - legal (encouraged even) to take address of stack variable
509 - many more
510
511 Plus some big ones...
512
513 * Bigger things
514
515 Some elements of Go step farther from C, even C++ and Java:
516
517 - concurrency
518 - garbage collection
519 - interface types
520 - reflection
521 - type switches
522
523 * Concurrency
524
525 * Concurrency
526
527 Important to modern computing environment.
528 Not well served by C++ or even Java.
529
530 Go embodies a variant of CSP with first-class channels.
531
532 Why CSP?
533
534 - The rest of the language can be ordinary and familiar.
535
536 Must be able to couple concurrency with computation.
537
538 Example: concurrency and cryptography.
539
540 * CSP is practical
541
542 For a web server, the canonical Go program, the model is a great fit.
543
544 Go _enables_ simple, safe concurrent programming.
545 It doesn't _forbid_ bad programming.
546
547 Focus on _composition_ of regular code.
548
549 Caveat: not purely memory safe; sharing is legal.
550 Passing a pointer over a channel is idiomatic.
551
552 Experience shows this is a practical design.
553
554 * Garbage collection
555
556 * The need for garbage collection
557
558 Too much programming in C and C++ is about memory allocation.
559 But also the design revolves too much about memory management.
560 Leaky abstractions, leaky dependencies.
561
562 Go has garbage collection, only.
563
564 Needed for concurrency: tracking ownership too hard otherwise.
565 Important for abstraction: separate behavior from resource management.
566 A key part of scalability: APIs remain local.
567
568 Use of the language is much simpler because of GC.
569 Adds run-time cost, latency, complexity to the implementation.
570
571 Day 1 design decision.
572
573 * Garbage collection in Go
574
575 A garbage-collected systems language is heresy!
576 Experience with Java: Uncontrollable cost, too much tuning.
577
578 But Go is different.
579 Go lets you limit allocation by controlling memory layout.
580
581 Example:
582
583 type X struct {
584 a, b, c int
585 buf [256]byte
586 }
587
588 Example: Custom arena allocator with free list.
589
590 * Interior pointers
591
592 Early decision: allow interior pointers such as `X.buf` from previous slide.
593
594 Tradeoff: Affects which GC algorithms that can be used, but in return reduces pressure on the collector.
595
596 Gives the _programmer_ tools to control GC overhead.
597
598 Experience, compared to Java, shows it has significant effect on memory pressure.
599
600 GC remains an active subject.
601 Current design: parallel mark-and-sweep.
602 With care to use memory wisely, works well in production.
603
604 * Interfaces
605
606 Composition not inheritance
607
608 * Object orientation and big software
609
610 Go is object-oriented.
611 Go does not have classes or subtype inheritance.
612
613 What does this mean?
614
615 * No type hierarchy
616
617 O-O is important because it provides uniformity of interface.
618 Outrageous example: the Plan 9 kernel.
619
620 Problem: subtype inheritance encourages _non-uniform_ interfaces.
621
622 * O-O and program evolution
623
624 Design by type inheritance oversold.
625 Generates brittle code.
626 Early decisions hard to change, often poorly informed.
627 Makes every programmer an interface designer.
628 (Plan 9 was built around a single interface everything needed to satisfy.)
629
630 Therefore encourages overdesign early on: predict every eventuality.
631 Exacerbates the problem, complicates designs.
632
633 * Go: interface composition
634
635 In Go an interface is _just_ a set of methods:
636
637 type Hash interface {
638 Write(p []byte) (n int, err error)
639 Sum(b []byte) []byte
640 Reset()
641 Size() int
642 BlockSize() int
643 }
644
645 No `implements` declaration.
646 All hash implementations satisfy this implicitly. (Statically checked.)
647
648 * Interfaces in practice: composition
649
650 Tend to be small: one or two methods are common.
651
652 Composition falls out trivially. Easy example, from package `io`:
653
654 type Reader interface {
655 Read(p []byte) (n int, err error)
656 }
657
658 `Reader` (plus the complementary `Writer`) makes it easy to chain:
659
660 - files, buffers, networks, encryptors, compressors, GIF, JPEG, PNG, ...
661
662 Dependency structure is not a hierarchy; these also implement other interfaces.
663
664 Growth through composition is _natural_, does not need to be pre-declared.
665
666 And that growth can be _ad_hoc_ and linear.
667
668 * Compose with functions, not methods
669
670 Hard to overstate the effect that Go's interfaces have on program design.
671
672 One big effect: functions with interface arguments.
673
674 func ReadAll(r io.Reader) ([]byte, error)
675
676 Wrappers:
677
678 func LoggingReader(r io.Reader) io.Reader
679 func LimitingReader(r io.Reader, n int64) io.Reader
680 func ErrorInjector(r io.Reader) io.Reader
681
682 The designs are nothing like hierarchical, subtype-inherited methods.
683 Much looser, organic, decoupled, independent.
684
685 * Errors
686
687 * Error handling
688
689 Multiple function return values inform the design for handling errors.
690
691 Go has no `try-catch` control structures for exceptions.
692 Return `error` instead: built-in interface type that can "stringify" itself:
693
694 type error interface { Error() string }
695
696 Clear and simple.
697
698 Philosophy:
699
700 Forces you think about errors—and deal with them—when they arise.
701 Errors are _normal_. Errors are _not_exceptional_.
702 Use the existing language to compute based on them.
703 Don't need a sublanguage that treats them as exceptional.
704
705 Result is better code (if more verbose).
706
707 * (OK, not all errors are normal. But most are.)
708
709 .image splash/fire.jpg
710
711 * Tools
712
713 * Tools
714
715 Software engineering requires tools.
716
717 Go's syntax, package design, naming, etc. make tools easy to write.
718
719 Standard library includes lexer and parser; type checker nearly done.
720
721 * Gofmt
722
723 Always intended to do automatic code formatting.
724 Eliminates an entire class of argument.
725 Runs as a "presubmit" to the code repositories.
726
727 Training:
728
729 - The community has always seen `gofmt` output.
730
731 Sharing:
732
733 - Uniformity of presentation simplifies sharing.
734
735 Scaling:
736
737 - Less time spent on formatting, more on content.
738
739 Often cited as one of Go's best features.
740
741 * Gofmt and other tools
742
743 Surprise: The existence of `gofmt` enabled _semantic_ tools:
744 Can rewrite the tree; `gofmt` will clean up output.
745
746 Examples:
747
748 - `gofmt`-r`'a[b:len(a)]`->`a[b:]'`
749 - `gofix`
750
751 And good front-end libraries enable ancillary tools:
752
753 - `godoc`
754 - `go`get`, `go`build`, etc.
755 - `api`
756
757 * Gofix
758
759 The `gofix` tool allowed us to make sweeping changes to APIs and language features leading up to the release of Go 1.
760
761 - change to map deletion syntax
762 - new time API
763 - many more
764
765 Also allows us to _update_ code even if the old code still works.
766
767 Recent example:
768
769 Changed Go's protocol buffer implementation to use getter functions; updated _all_ Google Go code to use them with `gofix`.
770
771 * Conclusion
772
773 * Go at Google
774
775 Go's use is growing inside Google.
776
777 Several big services use it:
778
779 - golang.org
780 - youtube.com
781 - dl.google.com
782
783 Many small ones do, many using Google App Engine.
784
785 * Go outside Google
786
787 Many outside companies use it, including:
788
789 - BBC Worldwide
790 - Canonical
791 - Heroku
792 - Nokia
793 - SoundCloud
794
795 * What's wrong?
796
797 Not enough experience yet to know if Go is truly successful.
798 Not enough big programs.
799
800 Some minor details wrong. Examples:
801
802 - declarations still too fussy
803 - `nil` is overloaded
804 - lots of library details
805
806 `Gofix` and `gofmt` gave us the opportunity to fix many problems, ranging from eliminating semicolons to redesigning the `time` package.
807 But we're still learning (and the language is frozen for now).
808
809 The implementation still needs work, the run-time system in particular.
810
811 But all indicators are positive.
812
813 * Summary
814
815 Software engineering guided the design.
816 But a productive, fun language resulted because that design enabled productivity.
817
818 Clear dependencies
819 Clear syntax
820 Clear semantics
821 Composition not inheritance
822 Simplicity of model (GC, concurrency)
823 Easy tooling (the `go` tool, `gofmt`, `godoc`, `gofix`)
824
825 * Try it!
826
827 .link / go.dev
828
829 .image splash/appenginegophercolor.jpg
830
831
View as plain text