Text file
talks/2015/how-go-was-made.slide
1 How Go was made
2 GopherCon Closing Keynote
3 9 Jul 2015
4
5 Andrew Gerrand
6 adg@golang.org
7
8
9 * Video
10
11 A video of this talk was recorded at GopherCon in Denver.
12
13 .link https://www.youtube.com/watch?v=0ht89TxZZnk Watch the talk on YouTube
14
15
16 * This talk
17
18 How was Go made?
19
20 What was its development process?
21
22 How has the process changed?
23
24 What is its future?
25
26 .image how-go-was-made/gopherswrench.jpg 300 _
27
28
29 * The early days
30
31
32 * Three friends
33
34 Robert Griesemer, Rob Pike, and Ken Thompson's thought experiment:
35
36 "What should a modern, practical programming language look like?"
37
38
39 * The early process
40
41 Design discussions by mail and in-person.
42
43 Consensus-driven:
44 A feature was accepted only when *all* the three authors agreed it necessary.
45
46 Most proposed changes were rejected.
47
48
49 * The initial spec
50
51 The first artifact of Go was the language specification.
52
53 The first commit (now git hash `18c5b48`) was a draft of that spec:
54
55 Author: Robert Griesemer <gri@golang.org>
56 Date: Sun Mar 2 20:47:34 2008 -0800
57
58 Go spec starting point.
59
60 Go's entire history is preserved in the core reposistory.
61
62
63 * The first Go program
64
65 The first Go program was a Prime Sieve, and was included in the spec.
66
67 func Filter(in *chan< int, out *chan> int, prime int) {
68 for {
69 i := <in; // Receive value of new variable 'i' from 'in'.
70 if i % prime != 0 {
71 >out = i // Send 'i' to channel 'out'.
72 }
73 }
74 }
75
76 func Sieve() {
77 ch := new(chan int); // Create a new channel.
78 go Generate(ch); // Start Generate() as a subprocess.
79 for {
80 prime := <ch;
81 printf("%d\n", prime);
82 ch1 := new(chan int);
83 go Filter(ch, ch1, prime);
84 ch = ch1
85 }
86 }
87
88
89 * Version control
90
91 Subversion was the project's first version control system. (no code review!)
92 The trio made 400 commits to Subversion.
93
94 The last Subversion commit (git hash `777ee7`, 21 July 2008) contained:
95
96 - the spec,
97 - a Go compiler (linux/darwin amd64),
98 - a few packages (fmt, rand, and math),
99 - some test programs.
100
101 `test/helloworld.go`:
102
103 package main
104
105 func main() {
106 print "hello, world\n";
107 }
108
109
110 * Getting serious
111
112 In July 2008 the project migrated from Subversion to Perforce,
113 to use Google's excellent code review system.
114
115 The team grew:
116
117 - Russ Cox joined in August 2008,
118 - Ian Lance Taylor joined in September,
119 - ~20 other Googlers got involved in their 20% time.
120
121
122 * Early changes
123
124 It was easy to make changes in those days.
125 Everyone knew each other.
126 There were ~0 users.
127 All the source code in one place.
128
129 Workflow informal, but design-oriented.
130
131
132 * Waiting for good design (1/2)
133
134 Many problems don't have obvious solutions.
135
136 The team were prepared to wait until they found the correct design.
137
138 For example:
139
140 The `fmt` package was written long before `reflect`.
141 Before then, `fmt` had an awkward chaining API:
142
143 fmt.New().s("i = ").d(i).putnl()
144
145 Now:
146
147 fmt.Println("i = ", i)
148
149
150 * Waiting for good design (2/2)
151
152 Another example:
153
154 Slices took more than a year to figure out.
155 Before then, there were "open arrays" but they were awkward.
156
157 `src/lib/container/vector.go` @ `f4dcf51`:
158
159 // BUG: workaround for non-constant allocation.
160 // i must be a power of 10.
161 func Alloc(i int) *[]Element {
162 switch i {
163 case 1:
164 return new([1]Element);
165 case 10:
166 return new([10]Element);
167 case 100:
168 return new([100]Element);
169 case 1000:
170 return new([1000]Element);
171 }
172 print "bad size ", i, "\n";
173 panic "not known size\n";
174 }
175
176
177 * An early language change
178
179 `chan` and `map` were originally spelled `*chan` and `*map`.
180
181 The team had a meeting to discuss removing the asterisks.
182
183 Russ implemented the compiler change (`dc7b2e9`),
184 and updated ~every Go file in existence (`08ca30b`).
185
186 .code how-go-was-made/mapchan.diff
187
188 This rapid approach works well at a small scale.
189
190
191 * Toward open source
192
193 In mid-2009 the team prepared for the open source release.
194
195 Moved to their third version control system:
196 Mercurial, with Rietveld for code review.
197 # (We would go on to use these tools for five years.)
198
199 The last Perforce commit (`9e96f25`, Oct 29) contained
200
201 - a mature spec and compiler,
202 - a broad standard library (82 packages),
203 - gofmt, godoc, cgo,
204 - the testing framework,
205 - the FAQ, Effective Go, and more.
206
207
208 * Open source
209
210
211 * The release
212
213 10 November 2009: Go was released. (`78c47c3`)
214
215 .image how-go-was-made/website.png 500 _
216
217
218 * The response
219
220 There was a huge response to the release. The team was overwhelmed.
221
222 People sent changes from day one. The first non-Googler change:
223
224 commit 022e3ae2659491e519d392e266acd86223a510f4
225 Author: Kevin Ballard <kevin@sb.org>
226 Date: Tue Nov 10 20:04:14 2009 -0800
227
228 Fix go-mode.el to work on empty buffers
229
230 Fixes #8.
231
232 R=agl, agl1, rsc
233 https://golang.org/cl/153056
234
235 In the first month, 32 people from outside Google contributed to Go.
236
237 The contribution process worked.
238
239
240 * Changes after the release
241
242 The team continued to make big changes after the release.
243
244 But the process was now different, as they now had a community.
245
246
247 * The first major public change (1/2)
248
249 On December 9th, Rob Pike sent the first public change proposal.
250
251 He proposed to remove semicolons at line endings.
252 [[/s/semicolon-proposal][go.dev/s/semicolon-proposal]]
253
254 The proposal was a "design doc" that included:
255
256 - background,
257 - rationale,
258 - a formal specification of the change,
259 - an implementation plan, and
260 - examples (including a copy of the Prime Sieve program without semicolons).
261
262
263 * The first major public change (2/2)
264
265 The design doc was shared with the community for feedback.
266
267 "Please read the proposal and think about its consequences.
268 We're pretty sure it makes the language nicer to use and sacrifices almost nothing in precision or safety."
269
270 The response was positive.
271
272 The proposal was implemented.
273
274 Most of the changes were made mechanically (thanks `gofmt`)).
275
276 This would be the shape of things to come.
277
278
279 * Curating change
280
281 * Curating change
282
283 The team was careful to curate the changes that made it into the language.
284 (Remember: "Wait for good design.")
285
286 There were many proposed changes and additions to Go.
287
288 As before, many more were declined than accepted.
289
290 Some fit with the project's goals, others did not.
291
292 An early accepted proposal:
293
294 - Make the upper and lower bounds of slice operations optional. (`x[lo:]`, `x[:hi]`)
295
296 And a declined proposal:
297
298 - Allow negative indices in slice operations. (`x[-n]` `==` `x[len(x)-n]`)
299
300
301 * Communicating goals
302
303 At first, we did a poor job explaining the project's goals and development process.
304
305 This caused frustration. ("Why don't they accept my suggestions?")
306
307 It took us a while to articulate it:
308 Rob Pike's [[/talks/2012/splash.article][talk]] in October 2012
309 "Go at Google: Language Design in the Service of Software Engineering"
310 was the first thorough explanation of Go's _raison_d'être_.
311
312 I wish we'd had this document written in 2009.
313 (But we couldn't have written it then.)
314
315 Read it, if you haven't already.
316
317
318 * A thought on openness
319
320 What if Go has been open source from day one?
321
322 It may have been easier for the public to understand the project.
323
324 But it's not that simple:
325
326 Ideas are fragile in their early stages; they need to be nurtured before exposed to the world.
327
328
329 * Managing change
330
331 * Weekly snapshots
332
333 Attempt to keep everyone in sync.
334
335 - Apply a Mercurial tag to a specific, stable revision.
336 - Announce to user mailing list with detailed changelog.
337
338 Great for early adopters and core developers.
339
340
341 * Problems with weeklies
342
343 Contributors work at tip; users sync to weeklies.
344
345 Burden on users:
346
347 - annoying to update weekly,
348 - painful to update less often.
349
350 Version skew results because users are at different weeklies.
351
352 Skew fragments the community and slows adoption.
353
354
355 * Formal release process
356
357 March 2011: introduced releases every 1-2 months.
358
359 - Pick the most stable of the past few snapshots and tag it.
360 - Announce with abridged "must read" release notes.
361
362 Keeps the community more in sync. Reduces churn.
363
364 Popular with users.
365
366 But skew still prevalent: adventurers and core devs still use weeklies (or tip!).
367
368
369 * Introducing Gofix
370
371 A tool to mechanically update code to accommodate language and library changes.
372
373 gofix prog.go
374
375 Announced in May 2011.
376
377 Eases the burden of staying current.
378 Release notes now mostly say "run gofix."
379
380 Not a sed script. Works on the AST.
381
382
383 * A gofix example
384
385 The `reflect` API was completely redesigned in 2011.
386 Gofix made most of the changes:
387
388 .image how-go-was-made/reflect1.png _ 1000
389 .image how-go-was-made/reflect3.png _ 1000
390 .image how-go-was-made/reflect2.png _ 1000
391
392
393 * Versioning issues persist
394
395 Gofix is no panacea.
396
397 As the root of the dependency graph, a programming language can suffer acutely from version skew.
398
399 The fundamental issue remains:
400 Code you write today may not compile tomorrow.
401
402 Some companies unwilling to bet on Go as they saw it as unstable.
403
404
405 * A need for stability
406
407 Gofix makes changes very easy, and also makes it easy to experiment.
408 But it can't do everything.
409
410 Priorities: If change is easy, what change is important?
411
412 Wanted to make major changes to the language and libraries,
413 but this requires planning.
414
415 Decision: design and implement a stable version of Go, its libraries, and its tools.
416
417
418 * Go 1
419
420 * What is Go 1?
421
422 A specification of the language and libraries that will be supported for years.
423
424 Available as downloadable binary packages.
425
426 An opportunity to:
427
428 - fix minor language irritations,
429 - fix inconsistencies in the standard library,
430 - focus on bug fixing and cleaning up TODOs,
431 - design and build a strong build tool set (get rid of make),
432 - bring Windows support up to par.
433
434 Polish and refine, not redesign.
435
436
437 * Planning Go 1
438
439 The team at Google prepared a detailed design document.
440
441 Implemented (but did not commit) many of the proposed changes.
442
443 Met for a week to discuss and refine the document (October 2011).
444
445 Presented the document to the community for discussion.
446
447 Community feedback essential in refining the document.
448
449 .link /blog/preview-of-go-version-1 go.dev/blog/preview-of-go-version-1
450
451
452 * An example: errors (1/5)
453
454 Before Go 1, there was no error type.
455 Instead, the `os` package provided an `Error` type:
456
457 package os
458
459 type Error interface {
460 String() string
461 }
462
463 References to `os.Error` were ubiquitous.
464
465 package io
466
467 type Reader interface {
468 Read(p []byte) (n int, err os.Error)
469 }
470
471 type Closer interface {
472 Close() os.Error
473 }
474
475
476
477 * An example: errors (2/5)
478
479 Before the Go 1 meeting, Russ raised the issue in the design document:
480
481 .image how-go-was-made/errors-issue.png 500 _
482
483
484 * An example: errors (3/5)
485
486 At the meeting the team discussed the issue,
487 Russ presented data from his experiments with the change,
488 and we made a tentative decision:
489
490 .image how-go-was-made/errors-discussion.png 350 _
491
492
493 * An example: errors (4/5)
494
495 On the list, the community made some keen suggestions:
496
497 .image how-go-was-made/errors-rog.png 500 _
498
499
500 * An example: errors (5/5)
501
502 Many of those suggestions were rolled into the Go 1 design document:
503
504 .image how-go-was-made/errors-final.png 500 _
505
506
507 * Implementing Go 1
508
509 Create many new issues on the tracker.
510
511 Contributors nominate themselves to address specific issues.
512
513 Stop developing new features; prioritize stability.
514
515
516 * The success of Go 1
517
518 The Go 1 release in March 2012 heralded a new era for the project.
519
520 Users appreciated the stability. Huge uptick in community growth.
521
522 .image how-go-was-made/trends.png
523
524 Contributors focused on implementation, tools, and ecosystem.
525
526
527 * The release cycle
528
529 The next release was Go 1.1, more than a year after Go 1.
530
531 This is too long; switch to a 6-month release cycle.
532
533 Stuck to this plan for 1.2, 1.3, and 1.4, and we're (almost) on track for 1.5.
534
535
536 * The future of change
537
538
539 * A question
540
541 Go's development process emphasizes up-front design.
542
543 The project has ~450 contributors and many committers from outside Google.
544
545 But most change proposals
546 (despite being driven by community feedback)
547 were made by the Go team at Google.
548
549 Why is this?
550
551
552 * Contribution guidelines
553
554 The Go contribution guidelines are thorough on the code review process.
555
556 But on design, just this:
557
558 "Before undertaking to write something new for the Go project, send mail to the mailing list to discuss what you plan to do."
559
560 Does this describe the project's approach to design? Only superficially.
561
562 Successful proposals include design docs
563 that discuss rationale, tradeoffs, and implementation.
564
565 There is a gap in our documented process.
566
567
568 * Go Change Proposal Process: a proposal
569
570 Brad Fitzpatrick and I recently proposed a formal Change Proposal Process.
571
572 Its goals:
573
574 - Make the design process clear to new contributors.
575 - Commit to timely evaluations of proposals.
576 - Build a recorded history of proposals.
577
578
579 * How does it work?
580
581 The new change process, in brief:
582
583 - File an issue.
584 - The contributors triage the issue.
585 - Write a design document (following a template).
586 - The contributors review the document.
587 - If the proposal is accepted, implement it.
588
589 [[/s/proposal-process][go.dev/s/proposal-process]]
590
591
592 * An experimental process
593
594 The new process is an experiment.
595
596 We're still discussing exactly how it should work.
597
598 I hope it will make Go's design process more accessible to the community.
599
600
601 * Conclusion
602
603 Go's design-driven process has served us well.
604
605 But as Go is more widely used,
606 the community needs a larger role in shaping its future.
607
608 With your help, we can make Go's next 5 years more spectacular than the last.
609
610 .image how-go-was-made/5years.png 350 _
611
View as plain text