How Go was made GopherCon Closing Keynote 9 Jul 2015 Andrew Gerrand adg@golang.org * Video A video of this talk was recorded at GopherCon in Denver. .link https://www.youtube.com/watch?v=0ht89TxZZnk Watch the talk on YouTube * This talk How was Go made? What was its development process? How has the process changed? What is its future? .image how-go-was-made/gopherswrench.jpg 300 _ * The early days * Three friends Robert Griesemer, Rob Pike, and Ken Thompson's thought experiment: "What should a modern, practical programming language look like?" * The early process Design discussions by mail and in-person. Consensus-driven: A feature was accepted only when *all* the three authors agreed it necessary. Most proposed changes were rejected. * The initial spec The first artifact of Go was the language specification. The first commit (now git hash `18c5b48`) was a draft of that spec: Author: Robert Griesemer Date: Sun Mar 2 20:47:34 2008 -0800 Go spec starting point. Go's entire history is preserved in the core reposistory. * The first Go program The first Go program was a Prime Sieve, and was included in the spec. func Filter(in *chan< int, out *chan> int, prime int) { for { i := out = i // Send 'i' to channel 'out'. } } } func Sieve() { ch := new(chan int); // Create a new channel. go Generate(ch); // Start Generate() as a subprocess. for { prime := Date: Tue Nov 10 20:04:14 2009 -0800 Fix go-mode.el to work on empty buffers Fixes #8. R=agl, agl1, rsc https://golang.org/cl/153056 In the first month, 32 people from outside Google contributed to Go. The contribution process worked. * Changes after the release The team continued to make big changes after the release. But the process was now different, as they now had a community. * The first major public change (1/2) On December 9th, Rob Pike sent the first public change proposal. He proposed to remove semicolons at line endings. [[/s/semicolon-proposal][go.dev/s/semicolon-proposal]] The proposal was a "design doc" that included: - background, - rationale, - a formal specification of the change, - an implementation plan, and - examples (including a copy of the Prime Sieve program without semicolons). * The first major public change (2/2) The design doc was shared with the community for feedback. "Please read the proposal and think about its consequences. We're pretty sure it makes the language nicer to use and sacrifices almost nothing in precision or safety." The response was positive. The proposal was implemented. Most of the changes were made mechanically (thanks `gofmt`)). This would be the shape of things to come. * Curating change * Curating change The team was careful to curate the changes that made it into the language. (Remember: "Wait for good design.") There were many proposed changes and additions to Go. As before, many more were declined than accepted. Some fit with the project's goals, others did not. An early accepted proposal: - Make the upper and lower bounds of slice operations optional. (`x[lo:]`, `x[:hi]`) And a declined proposal: - Allow negative indices in slice operations. (`x[-n]` `==` `x[len(x)-n]`) * Communicating goals At first, we did a poor job explaining the project's goals and development process. This caused frustration. ("Why don't they accept my suggestions?") It took us a while to articulate it: Rob Pike's [[/talks/2012/splash.article][talk]] in October 2012 "Go at Google: Language Design in the Service of Software Engineering" was the first thorough explanation of Go's _raison_d'être_. I wish we'd had this document written in 2009. (But we couldn't have written it then.) Read it, if you haven't already. * A thought on openness What if Go has been open source from day one? It may have been easier for the public to understand the project. But it's not that simple: Ideas are fragile in their early stages; they need to be nurtured before exposed to the world. * Managing change * Weekly snapshots Attempt to keep everyone in sync. - Apply a Mercurial tag to a specific, stable revision. - Announce to user mailing list with detailed changelog. Great for early adopters and core developers. * Problems with weeklies Contributors work at tip; users sync to weeklies. Burden on users: - annoying to update weekly, - painful to update less often. Version skew results because users are at different weeklies. Skew fragments the community and slows adoption. * Formal release process March 2011: introduced releases every 1-2 months. - Pick the most stable of the past few snapshots and tag it. - Announce with abridged "must read" release notes. Keeps the community more in sync. Reduces churn. Popular with users. But skew still prevalent: adventurers and core devs still use weeklies (or tip!). * Introducing Gofix A tool to mechanically update code to accommodate language and library changes. gofix prog.go Announced in May 2011. Eases the burden of staying current. Release notes now mostly say "run gofix." Not a sed script. Works on the AST. * A gofix example The `reflect` API was completely redesigned in 2011. Gofix made most of the changes: .image how-go-was-made/reflect1.png _ 1000 .image how-go-was-made/reflect3.png _ 1000 .image how-go-was-made/reflect2.png _ 1000 * Versioning issues persist Gofix is no panacea. As the root of the dependency graph, a programming language can suffer acutely from version skew. The fundamental issue remains: Code you write today may not compile tomorrow. Some companies unwilling to bet on Go as they saw it as unstable. * A need for stability Gofix makes changes very easy, and also makes it easy to experiment. But it can't do everything. Priorities: If change is easy, what change is important? Wanted to make major changes to the language and libraries, but this requires planning. Decision: design and implement a stable version of Go, its libraries, and its tools. * Go 1 * What is Go 1? A specification of the language and libraries that will be supported for years. Available as downloadable binary packages. An opportunity to: - fix minor language irritations, - fix inconsistencies in the standard library, - focus on bug fixing and cleaning up TODOs, - design and build a strong build tool set (get rid of make), - bring Windows support up to par. Polish and refine, not redesign. * Planning Go 1 The team at Google prepared a detailed design document. Implemented (but did not commit) many of the proposed changes. Met for a week to discuss and refine the document (October 2011). Presented the document to the community for discussion. Community feedback essential in refining the document. .link /blog/preview-of-go-version-1 go.dev/blog/preview-of-go-version-1 * An example: errors (1/5) Before Go 1, there was no error type. Instead, the `os` package provided an `Error` type: package os type Error interface { String() string } References to `os.Error` were ubiquitous. package io type Reader interface { Read(p []byte) (n int, err os.Error) } type Closer interface { Close() os.Error } * An example: errors (2/5) Before the Go 1 meeting, Russ raised the issue in the design document: .image how-go-was-made/errors-issue.png 500 _ * An example: errors (3/5) At the meeting the team discussed the issue, Russ presented data from his experiments with the change, and we made a tentative decision: .image how-go-was-made/errors-discussion.png 350 _ * An example: errors (4/5) On the list, the community made some keen suggestions: .image how-go-was-made/errors-rog.png 500 _ * An example: errors (5/5) Many of those suggestions were rolled into the Go 1 design document: .image how-go-was-made/errors-final.png 500 _ * Implementing Go 1 Create many new issues on the tracker. Contributors nominate themselves to address specific issues. Stop developing new features; prioritize stability. * The success of Go 1 The Go 1 release in March 2012 heralded a new era for the project. Users appreciated the stability. Huge uptick in community growth. .image how-go-was-made/trends.png Contributors focused on implementation, tools, and ecosystem. * The release cycle The next release was Go 1.1, more than a year after Go 1. This is too long; switch to a 6-month release cycle. Stuck to this plan for 1.2, 1.3, and 1.4, and we're (almost) on track for 1.5. * The future of change * A question Go's development process emphasizes up-front design. The project has ~450 contributors and many committers from outside Google. But most change proposals (despite being driven by community feedback) were made by the Go team at Google. Why is this? * Contribution guidelines The Go contribution guidelines are thorough on the code review process. But on design, just this: "Before undertaking to write something new for the Go project, send mail to the mailing list to discuss what you plan to do." Does this describe the project's approach to design? Only superficially. Successful proposals include design docs that discuss rationale, tradeoffs, and implementation. There is a gap in our documented process. * Go Change Proposal Process: a proposal Brad Fitzpatrick and I recently proposed a formal Change Proposal Process. Its goals: - Make the design process clear to new contributors. - Commit to timely evaluations of proposals. - Build a recorded history of proposals. * How does it work? The new change process, in brief: - File an issue. - The contributors triage the issue. - Write a design document (following a template). - The contributors review the document. - If the proposal is accepted, implement it. [[/s/proposal-process][go.dev/s/proposal-process]] * An experimental process The new process is an experiment. We're still discussing exactly how it should work. I hope it will make Go's design process more accessible to the community. * Conclusion Go's design-driven process has served us well. But as Go is more widely used, the community needs a larger role in shaping its future. With your help, we can make Go's next 5 years more spectacular than the last. .image how-go-was-made/5years.png 350 _