Transitioning the Go SIMD generators to internal/spec

This plan lays out how to transition the myriad generators and hand-written sources of the SIMD API from disjoint, manually-enforced consistency, to a single underlying executable API specification that provides all method signatures, documentation, and checkable semantics.

See README.md for background on the SIMD packages, the “one name, one meaning” rule, the generators, how simdgen works, what spec and specgen are, and how to run things.

Table of contents

• Part 1: Decisions and rationale
• Part 2: Task graph
• Part 3: Task details
• Part 4: Risks and unknowns
• Appendix A: Regenerating the two generated files

The plan in brief

internal/spec — generic Go functions describing each operation — becomes the single hand-written description of the SIMD API. Today the same operation is described in six places, and they have already drifted from each other (§1.5).

Three things have to happen, and they are largely independent:

1. Build the specfill checker. One AST pass that injects spec’s docs into generated and hand-written source and verifies signatures against spec. Types can be enforced almost at once — SPEC-STATS.txt §2 shows zero type-level mismatches between spec and the shipped API.
2. Write the spec functions (spec-common and spec-all). Coverage is in SPEC-STATS.txt §1. This is the bottleneck the other tracks draw on.
3. Retire the old inputs: the other doc sources, then categories.yaml, at which point the unifier no longer defines the API — it only expands one arch entry into the machine instructions matching it.

Behavioral conformance against internal/simdref (track C) runs in parallel. Beyond its own ci-audit, it depends only on spec coverage and on the width and impl-defined decisions.

Monitoring plan progress

Progress on this transition is tracked in two ways:

Task completion is tracked directly in this document via the [ ] checkboxes on each task and risk. The task table in part 2 summarizes all task completion in one place.

SPEC-TASKS.dot shows a visual representation of the task graph and completed tasks. It is generated directly from this document by running go run ./cmd/spectasks -w from _gen. This generator also checks the consistency of this doc, and the test in ./cmd/spectasks enforces doc consistency in CI. Rerun the generator after modifying any task metadata.

Measurements live in SPEC-STATS.txt, which tracks spec coverage of the API, signature and name agreement, documentation sources, and retirement of the old inputs. It is generated by running go run ./cmd/specstats -w from _gen. Every figure in that file is a fraction heading for 100% or a count heading for zero, tagged with the tasks that move it. Indented, untagged numbers are context breaking down the figure above them. Part 1 of this doc quotes measurements that motivated specific decisions, stamped with the CL at which they were measured; those are frozen and do not need refreshing.

How to implement a task

Part 2 tells you which task to pick up: the next one whose prerequisites are all ticked. SPEC-TASKS.dot draws the same thing if a picture helps. The commit sequences in Part 3 are guesses made before the work started; deviate when the code tells you to, but say so in the CL description.

Tasks and risks carry [ ] checkboxes. When you resolve one, tick the box, update the task text to reflect the reality of the implementation so it remains useful to reference when performing follow-on tasks (you can drop or summarize the exact commit sequence), and regenerate SPEC-TASKS.dot. Commit this as its own CL on top of the CLs that implement the task, and include justifications and reasoning for any changes to the task text in the commit message.

Key concepts and terms

specfill is the tool this plan builds first and then leans on. It is one AST pass over Go source that injects spec-derived doc comments and parameter and result names, and verifies signatures against spec. It exists as both a package that can be integrated into generators and a standalone tool:

• specdoc.Fill is the library entry point — Go source in, Go source out: func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error). If verification checks fail, it returns a *Report as an error. Generators call it on their gentools buffer before writing.
• cmd/specfill is the gentools-based command wrapping it, for running over hand-written files. Where this document writes specfill -w, that is the command; where it writes specfill, that is the library function. CI enforcement is a go test in this package.

These package and command names are suggestions; the split between library and command is not.

Target means one of amd64, arm64 (NEON), arm64 SVE, or wasm. See README.md.

Spec operation means one function in package spec. Each expands across element types, widths, and templated names into many API names and many (receiver, name) pairs, so it is the grain spec is authored at — and the only grain at which “implemented nowhere” means anything (SPEC-STATS.txt §5).

Operation (unqualified) means the API-level operation a name denotes, whichever of spec, categories.yaml, or generated source describes it. A categories.yaml entry is called an entry; one operation name can have several.

Granularity. Spec coverage is measured at three granularities: (target, receiver, name) — every exported declaration, counted once per build configuration; (receiver, name) — collapsed across targets, the primary denominator since spec describes an operation once; and name — collapsed across receivers too, coarser and misleading when a name exists on different receiver sets in spec and the API (SPEC-STATS.txt §5). SPEC-STATS.txt §1 reports all three. String and Len are excluded throughout (§1.7).

---

Part 1 — Decisions and rationale

These are the load-bearing decisions. Each was contested during design. The reasoning is recorded so it need not be re-litigated, while also making it easier to revisit decisions if new evidence arises.

Measurements in this section were taken at CL 831485 unless otherwise noted. They should NOT be updated unless decisions need revisiting.

1.1 We are not rewriting simdgen

The temptation is real: simdgen has 28 stub templates whose names encode arity, operand permutation, and immediate placement (op3_231Type1, op4_31, op2Imm8_II); 42 fields across RawOperation (23) and Operand (19); and 27 bail-outs whose messages begin “simdgen only supports” or “simdgen does not”. But the layer accounting says the mess is small and the value is large (at CL 831485; Go counts include _test.go; “signature derivation + emission” is simdgen/gen_*.go, godefs.go, arch.go, and types/operation.go):

The part that would be thrown away if unification were the wrong call is the ~500 messiest lines of signature derivation, against irreplaceable platform loaders several times that size.

A few thousand lines of hand-written YAML drive tens of thousands of lines of generated API and compiler backend across four targets. That ratio is good. What is not earning its keep is unification’s role as API definer — the categories.yaml ↔ go_GOARCH.yaml join — and this transition removes that role by construction. The unifier’s remaining job, expanding one arch entry into every matching concrete instruction variant against the ISA database, it does well.

1.2 Spec is not a unifier input

Feeding spec into unification via DefBuilder is easy and wrong. Spec’s structure would be flattened into the unstructured value language the moment it crossed the boundary, recreating the exact failure mode we are escaping. Instead, simdgen consumes []*specgen.Func as a typed Go table.

Furthermore, spec is inherently only aware of the structure of the Go SIMD API, but the inputs to the unifier are in platform terms. Much of the output of the unifier is information on how to map the platform structure into the Go SIMD API, so we can’t bring spec in until after that mapping has been processed.

1.3 What belongs in spec

The test is: is it a checkable claim about the spec’s own semantics?

• commutative — yes. Add(x,y) == Add(y,x) is testable against the executable body, and spec is the only place that can check it.
• noGenericOps, noTypes — no. Claims about generator output.
• constImm — no. A claim about x86 encoding.
• regexpTag, signed — no. Claims about the unifier.

The test is self-policing: anything that passes it falls out of the conformance testing we want anyway. Note it admits facts the compiler consumes — the wrong line to draw is “API vs compiler”; the right one is “intrinsic to the operation vs incidental to the mechanism or platform.”

For commutative, no name in categories.yaml currently needs platform-specific commutativity, so a function-level directive in spec is sufficient today. However, there may be corner cases we haven’t uncovered; notably, we have evidence that Min/Max on floats may not be commutative on all platforms. It may suffice to simply not mark these operations as commutative, and to document when an operation is not commutative for surprising reasons.

Applying the test to categories.yaml’s fields (at CL 831485; 217 entries, 195 unique operation names, 163 exported):

Task cat-rehome tackles most of the “transition plan” column. compiler-ops-impl handles the noGenericOps / noTypes entries.

1.4 specfill parses; it does not type-check

ops_amd64.go and types_amd64.go have minimal imports, and no exported signature in either package uses a qualified type, so go/parser is sufficient — no go/types, no packages.Load, no GOEXPERIMENT wrangling, no cross-module loading of GOROOT std packages.

This is better than a workaround. Type-checking sees one build configuration at a time and would make the cross-architecture comparison harder. Parsing all files regardless of build tag yields (Int32x4) Add from ops_amd64.go, ops_arm64.go, and ops_wasm.go in a single pass, which is precisely the comparison we want. apisum already does this, so it is proven on this API.

cmd/specstats (Appendix A) takes a different route to the same end: it loads syntax through go/packages once per build configuration, because it reports per-target figures. It still does not type-check, and specfill does not need per-configuration loading.

1.5 Two-pass doc injection, and why the “disadvantage” is the point

The motivation is doc divergence. At CL 831485 there were six independent documentation sources — categories.yaml, wasmgen inline strings, tmplgen inline strings, midway/comments.yaml, hand-written simd_emulated.go, and hand-written archsimd files — and none agreed with any other. All 90 (receiver, name) pairs documented in both simd and archsimd had different doc text, and within the portable package, simd_stubs.go (midway-generated) and simd_emulated.go (hand-written) had drifted on 40 of 336 mirrored methods. Add shipped with three independent wordings:

124 method names are produced by more than one generator — the easiest place for disagreement to creep in. simdgen accounts for the majority of declarations, but tmplgen, wasmgen, midway, and hand-written code own the rest, and those sets overlap — any instrument that only sees simdgen misses a substantial fraction of the problem (SPEC-STATS.txt §1 has the per-generator counts).

Generators pass their generated code to specfill, which checks signatures and injects docs before writing to disk; in the end state, generators emit declarations and implementation notes (§1.6) but no API prose of their own. The alternative — having specgen emit the whole declaration, as refgen does — was rejected because it would make agreement with spec true by construction and leave nothing to verify. A generator whose internal model of the API had drifted would be silently papered over. Because the template still produces the signature, specfill sees two independent witnesses to the same declaration and can compare them: docs in, signatures verified, coverage reported, one AST walk.

Consequences:

• specfill is an override, not a replacement. Generators keep emitting their current docs; specfill replaces the spec-owned paragraphs only when spec knows the operation. Fallback is therefore automatic and no flag day is required. Prose-less emission is the end state reached when a generator’s doc source empties, not a precondition.
• wasmgen and tmplgen need only a call before flush, plus making the notes they emit follow §1.6’s prefixes.
• Do not hook this into gentools.Files.Flush(). gentools is a general utility; coupling it to specgen would make every consumer depend on golang.org/x/tools/go/packages and a loadable spec tree. A narrow hook mechanism in gentools plus a few lines per generator keeps the boundary.

1.6 The doc-ownership convention

specfill rewrites part of an existing doc comment, so it needs a rule for which part it may touch, and running it twice must be idempotent. The obvious rule, “everything above the first bare //”, doesn’t work. At CL 831485 only 2,759 of 3,920 documented declarations have that separator. And spec’s own docs contain blank comment lines: Add’s is a summary, a //, then z[i] = x[i] + y[i]. Hence, using a bare // would not be idempotent.

The rule. A doc comment is spec-owned paragraphs, followed by “implementation notes”, followed by directive comments (//go:fix inline and similar). Spec-owned paragraphs describe the behavior of an operation. Implementation notes are details of how one target implements an operation. The sections must appear strictly in this order, and the tools will validate this.

An implementation note is a paragraph that starts with one of a fixed set of prefixes: Asm:, CPU Feature:, Emulated[,\n], Deprecated:, Performance:, or Note:. The match is exact and case-sensitive. Only the first line of a paragraph is checked, so a note can combine prefixes (Asm: VPADDD, CPU Feature: AVX) and wrap freely.

Checks. Strict ordering is necessary for idempotence, and the tools perform the following checks to ensure we maintain it: (a) In an existing comment, spec-owned paragraphs (those without an implementation note prefix) must appear strictly before implementation notes (those with an implementation note prefix), which must appear strictly before directive comments. (b) Spec’s docs may not contain a line starting with a note prefix, and spec stores them in canonical go/doc/comment form.

Rewrite. specfill rewrites a comment only when spec has an entry for the declaration and check (a) passes. The new comment is the spec doc, then a blank line if there are implementation notes, then those notes unchanged, then the directives unchanged. In every other case the comment is left alone. This is idempotent: by (b) the injected spec doc never looks like a note, so the next run splits the comment at the same place and produces the same bytes. Idempotence can’t catch text lost on the first run, though, so as we transition to spec, generated docs for newly covered operations must be reviewed carefully.

Why prefixes, not markers. The alternative was bracketing specfill’s region with directive comments (//spec:doc … //spec:end). Those are invisible in godoc and safe by default — no marker, nothing touched — but they add lines to every declaration and say nothing about why text is outside spec. Prefixes reuse the shape the notes already have, work the same in generated and hand-written files, and force each piece of non-spec text to say what kind of note it is. The cost is that unmarked text defaults to spec-owned, which is why every rewrite is reviewed as a diff, and why generators, once they emit no API prose of their own, may not emit spec-owned text at all (see specfill).

1.7 Errors versus tolerated gaps

At CL 831485, of the 782 (receiver, name) pairs declared for more than one target, not one had divergent parameter or result types — but 72 disagreed on parameter names and 16 on result names. So spec is preventive, not remedial: there is no backlog of type-level fixes, and incremental migration is not racing decay.

• Signature mismatch is an error. For an operation spec knows about, a disagreement is a bug in spec or in a generator.
• A missing spec function is tolerated until the transition completes.
• Parameter and result names are checked where the declaration names them (unnamed ones are filled from spec; §1.8), for two reasons: they appear in documentation and so must be consistent with the documentation’s source, and several past bugs have come from confused or swapped argument names.
• specfill considers exported declarations only. This is what makes the tolerated-miss list converge: all three ops_internal_*.go files (amd64, arm64, sve) contain zero exported declarations, so every compiler-only operation falls outside specfill’s view without needing a category. The permanent residue is small and enumerable: the String, Len, ToArch methods and xFromArch functions.

1.8 Named results

Spec names its results and its docs reference those names (z[i] = x[i] + y[i]). Almost nothing in the shipped API names its results (SPEC-STATS.txt §2). When spec’s doc becomes archsimd’s doc, the references to z in the docs won’t make sense unless we carry over spec’s result names as well.

Decision: where spec covers a declaration, its parameter and result names come from spec, so they agree with the doc that also comes from spec. Naming every result is not a goal: a name is worth having where a doc or a body uses it, and naming the rest blindly serves no one. Declarations spec does not cover keep whatever names they have until spec-all covers them.

specfill enforces this without touching bodies. It rewrites unnamed arguments/results in the API (including _) to match spec, and reports a mismatch error if both sides name an argument/result and the names disagree. For a bodyless declaration, all that matters is that the names agree with the doc. A body can refer to named arguments but not unnamed ones, so filling unnamed ones is safe, while named ones must be checked. SPEC-STATS.txt §2 counts the two cases separately, the same way.

1.9 Behavioral conformance is a parallel track

Behavioral conformance has no hard dependency on the API documentation track. Both consume the same input — spec coverage — independently, and they catch disjoint bug classes: a swapped argument in a new architecture’s Sub has an identical signature and identical docs, and only conformance sees it.

The order for bootstrapping conformance matters. Validate spec against amd64 first — it is the most mature target, and every failure is a spec bug unless proven otherwise. There may be bugs or hardware-specific corner cases in the amd64 implementation, but bugs in spec are far more likely. Once spec and amd64 agree, then flip: spec becomes the oracle and gates arm64, wasm, SVE, and any architecture added later. Getting this backwards means conformance failures against correct implementations, and the obvious move is then to “fix” the implementation.

Harness requirements implied by the bug classes we actually see: per-type edge-case inputs (0, ±1, MinInt, MaxInt, ±0, ±Inf, NaN, saturation boundaries) crossed with random vectors — uniform random will not reliably hit saturation or NaN corners; and explicit mask patterns, since an inverted mask sense passes random testing about half the time.

This is a path we have already gone down successfully. simd/internal/test_helpers carries per-type input vectors — float32s through uint8s, plus bools for masks — built by cycling a hand-picked list of exactly those corners (±0, ±Inf, quiet and signalling NaN, MaxFloat, and the 8- and 16-bit saturation edges) up to a fixed length. In archsimd/internal/simd_test/helpers_test.go, the forSlice/forSlicePair/forSliceTriple/forSlicePairMasked helpers walk windows over them for the cross-product, map1/map2/map3/mapCompare build the expected result elementwise — the same shape a spec body has — and grouped2 already handles the 128-bit grouping that gives doc-explore trouble. checkSlicesLogInput even takes a tolerance, which is a first cut at the vocabulary impl-defined has to settle.

conform-fixed need not reuse that code literally, but it should not start over either: what it adds is generating the cases from spec rather than hand-writing them per operation.

1.10 Progress is measured by name, not by normalization

apisum needs normalization because its job is to test whether names mean one thing. Once we enforce that they do, the method name is the equivalence class, exactly and for free: coverage is a set intersection, and cmd/specstats already computes it. No name normalization, and no complex pattern matching rules to maintain or throw away.

Caveat, worth knowing rather than fixing: templated-name families like LoadInt8x16, LoadInt8x32, etc, that all come from one spec function inflate the denominator of any coverage metrics. That is arguably correct for measuring API surface. Reporting both numbers — “spec has N functions covering M names” — is more informative than either alone.

---

Part 2 — Task graph

The 27 tasks fall into six tracks. Tracks are a way to divide the work among people and to make Part 3 navigable. They are not phases and don’t imply any ordering on tasks. Tasks in each track generally cluster in the task dependency graph, but there are inter-track dependencies.

Dependencies

A task is startable when every prerequisite is ticked. Hard dependencies are listed as Needs; soft dependencies — work is possible but wasteful or lower-value without them — as Prefers.

Tasks are named, not numbered, so that a task’s identity survives inserting another one next to it, and so that a CL saying “implements cat-rehome” still means something a year later. Ordering lives in the row order instead: the table is topologically sorted (both Needs and Prefers count as edges), with ties broken by track, so reading it top to bottom is a legal order to do the work in, and cmd/spectasks fails if an edit breaks that. Add a task by putting its row wherever its prerequisites and track allow.

Critical path. SPEC-TASKS.dot draws it in red: every task on some longest chain of Needs edges. Slipping any of them slips the whole project. cmd/spectasks derives it from the table, so it is not restated here.

Track B has no prerequisites, so it can start immediately and run to completion at its own pace — though gen-docs gates three of track E’s tasks. sig-refactor is independent of everything until sig-from-spec.

Milestones

Milestones carry no work of their own; they are the points worth announcing. A milestone is reached when its tasks are ticked, so there is no separate state to maintain and nothing here to keep in sync.

The graph

SPEC-TASKS.dot draws the two tables above; the rendered graph carries its own key. Track is node colour rather than a cluster, because a cluster has to be laid out contiguously and that fights the left-to-right dependency order the graph exists to show.

dot -Tsvg SPEC-TASKS.dot -o SPEC-TASKS.svg

It is generated, and its generator is also a checker. This document states its dependencies three times — Part 2’s table, Part 3’s Needs:/Prefers: lines, and Part 3’s Blocks: lines, which are the same edges reversed — so go test ./cmd/spectasks fails if they disagree, or if the graph has not been regenerated since. It also derives the critical path.

Start here

The startable tasks are the ones with no unticked Needs in the dependency table. For a team of one to three, this is a reasonable opening allocation, covering every task with no Needs at all:

• Strongest on spec’s type model: width, then convert-lo and mask-bits. These root the longest chain and cost the most if deferred.
• Owns API semantics: impl-defined and compiler-ops — pure decisions, no code, and everything downstream waits on them.
• Likes tooling: specfill, then fill-gen and named-results.
• Working alone on something large: sig-refactor. No prerequisites, verifiable byte-for-byte at every step.
• Cheap and worth doing early regardless: doc-explore and ci-audit.

What “done” looks like

After cat-delete, simd/internal/spec is the only hand-written description of the API: every operation’s name, signature, and documentation derives from it, in both public packages and on all four targets, and specfill fails the build if any of them drift. categories.yaml is gone. midway/comments.yaml documents only what spec does not describe: the vector types, VectorSize, String and Len. wasmgen and tmplgen no longer carry doc strings. Hand-written implementations remain hand-written, but their operation doc comments are generated.

simdgen still exists and is still unification-based — §1.1 explains why. What changes is its job: the unifier no longer defines the API, it only expands one go_GOARCH.yaml entry into all the concrete machine instructions matching it. go_GOARCH.yaml stays, and so do its quirk fields, because instruction irregularity is real and has to live somewhere. The compiler backend generation is untouched throughout.

---

Part 3 — Task details

Many completion criteria are “generated output is byte-identical”, checked with go run . -diff from _gen. That runs every generator, including simdgen for all three of its targets, so it needs the XED and ARM ISA data first — see fetch-xed.sh and fetch-arm64.sh. There is no -arch flag on the driver; -arch belongs to simdgen when run directly.

Track A — Decisions

Human judgment calls. None has prerequisites, and each blocks downstream work — which is why they lead. Each is small in code and large in consequence.

width — Resolve scalable width

Done: [x] · Needs: — · Blocks: spec-common, conform-scalable · Risk: risk-concrete-width

simd/internal/spec/types.go hardcodes scalableWidth = 4096, and simdref is built at that width, while archsimd’s SVE uses the runtime vl(). A conformance test of scalable types therefore compares vectors of different lengths. Spec’s own TODO flags this and it is unresolved.

4096 bits in particular causes trouble for operations that index into vectors, such as shuffles, which is why every platform other than RISC-V RVV is 2048 bits or less. Resolving this task may require deciding whether we’ll ever support vectors more than 2048 bits wide.

Note what is not wrong: spec’s type system was designed for scalable widths and handles them. The gap appears only when spec is used as a concrete conformance oracle, and the open question is narrow — whether spec can execute at a caller-chosen width without disturbing the generic type machinery.

This is first because the fix may change spec’s type model — making width a runtime parameter rather than a compile-time constant — which reaches every spec function body. Resolving it after a hundred bodies exist is far worse than resolving it now, while there are few (SPEC-STATS.txt §1).

Commits. (1) Write up the options with their consequences for Vec, makeVec, lanes, refgen, and the conformance harness; seen so far: parameterize width at spec execution, make simdref width-configurable at construction, or scope conformance to fixed widths initially and defer. (2) Implement the chosen approach in internal/spec. (3) Regenerate internal/simdref and confirm cmd/specls output is unchanged, or explain what changed.

Done when a scalable-typed operation can be executed through simdref at a width matching a chosen target, and the choice is documented in internal/spec/doc.go.

compiler-ops — Compiler-only ops decision

Done: [ ] · Needs: — · Blocks: compiler-ops-impl

Decide how the compiler-only entries — those that produce no exported declaration (SPEC-STATS.txt §4) — and the noTypes / noGenericOps switches are represented once categories.yaml is gone. This is the only hard blocker in the chain without an obvious answer, so make it early.

Note §1.7: because specfill is scoped to exported declarations, this does not block the checker or conformance tracks.

The design note must also say where the following go once categories.yaml is gone: the compiler-only entries’ documentation:, which feeds the doc comments of unexported methods in archsimd/ops_internal_*.go; commutative, signed and out on those entries; and the go_arm64.yaml use of noGenericOps. This documentation is strictly internal-facing, so it is held to a lower standard than user-facing API docs. It need not come from spec, go through specfill, or satisfy M2. Wherever it lands is acceptable as long as it is kept next to the compiler-only op’s other description.

Commits. One design note in this document or in internal/spec/doc.go, listing all compiler-only operations from _gen/simdgen (grep -h "go: [a-z]" ops/*/categories.yaml | sort -u; also the capitalized noTypes entries) and stating for each family where its description, documentation, and remaining fields live once categories.yaml is gone. Candidates: a //specgen:unexported directive; a fourth simdgen input holding compiler-only ops; or leaving them in go_GOARCH.yaml on the grounds that they are per-architecture lowering details rather than API.

Done when the decision is written down, including documentation and all remaining fields, and compiler-ops-impl could be implemented from it without further design work.

convert-lo — ExtendLo* / ConvertLo* shape and naming

Done: [ ] · Needs: — · Prefers: width · Blocks: spec-all · Risk: risk-spec-defects

Settle how the low-elements widening family is shaped and named, across both fixed-width and scalable vectors. An API design decision, not a mechanical fix. Three entangled problems:

1. Shape. ExtendLoLToZ declares x Vec[E,W] and z Vec[zE,W] — the same total width. The shipped API widens instead: Int8x16.ExtendLo8ToInt64() takes 128 bits and returns 512. Spec therefore misses 8 real declarations and invents width-preserving ones no platform has.
2. Naming. //specgen:name ExtendLo{zL}To{zE} embeds the result lane count, so the same conceptual operation gets a different name at each width. The API has settled on 128-bit receivers only, which hides this.
3. Scalable. The converts.go TODO already notes the literal zL cannot work for scalable vectors, and suggests dropping the number entirely since it is implied by the receiver and target types — e.g. Uint8x16.ExtendLo2ToUint64 is really the low eighth, not the low half, so the number is arguably misleading even where it works.

Any renaming here is API-visible and should be decided before spec-all migrates the Converts family, which is where the bulk of categories.yaml’s Converts documentation lands.

Commits. (1) Write up the options and their consequences for fixed-width and scalable vectors, including whether the lane count stays in the name. (2) Re-shape ExtendLoLToZ, and re-check ConvertLoLToZ, whose zL<xL constraint currently admits conversions no hardware offers. (3) If the API name changes, land the rename through the generators as its own CL, separate from the spec change.

Done when cmd/specls produces exactly the ExtendLo* declarations the API has, plus whatever maximalist extras are deliberate, and the scalable naming question is answered in internal/spec/doc.go.

mask-bits — Mask bitmap load/store shape and naming

Done: [ ] · Needs: — · Prefers: width · Blocks: spec-all · Risk: risk-spec-defects

Settle the scalable counterpart of fixed-width ToBits and MaskNxMFromBits (e.g. Mask8x16FromBits), which today are Mask*s.Store and LoadMask*s. Also settle what Store and ToBits mean on portable simd.Mask*s, whose midway docs currently describe element semantics.

The API declares Store(bits []uint16) only on SVE’s scalable masks (archsimd/types_sve.go), paired with LoadMask8s(bits []uint16) etc. It stores SVE predicate bits at byte granularity. No fixed-width mask has Store; fixed-width masks have ToBits() UintN instead (spec: MaskToBits, internal/spec/masks.go, constrained to FixedWidth). So Mask*s.Store has a different meaning from every other Store — a candidate violation of the “one name, one meaning” rule. Its bit layout is SVE’s representation, not a per-lane one, which arguably fails §1.3’s “claim about the operation’s own semantics” test.

Commits. (1) Write up the options and their consequences for fixed-width and scalable masks, and for the portable package. (2) Change spec to match. (3) If the API name changes, land the rename through the generators as its own CL.

Done when SPEC-STATS.txt §5 lists no mask Store defect, and the choice is recorded in internal/spec/doc.go.

impl-defined — Implementation-defined behavior policy

Done: [ ] · Needs: — · Blocks: validate-amd64 · Risk: risk-impl-defined-behavior

Decide what counts as implementation-defined behavior in the SIMD API, and how spec expresses it so conformance testing stays meaningful. Architectures legitimately differ on corners — the last bits of a float result, out-of-range shuffle indices, saturation edges, NaN propagation and signed zero. Spec’s math.go already carries a TODO on DotProductPairs overflow that names the problem and floats an “architecture dependent” designated panic as one possible answer.

Two halves, and both need answering:

1. What is implementation-defined? A per-operation API-semantics judgment. Costly in both directions: declare too much and conformance stops catching real bugs; declare too little and correct implementations fail against spec.
2. How is it written down? There are at least two major categories of implementation-defined behavior: exceptional-value cases (e.g., overflow behavior is fully undefined) and precision (all architectures roughly agree, but maybe not to the last digit). Spec bodies return exact values today, and there is no vocabulary for “any value here”, “within N ulp”, or “unspecified but deterministic per platform”. Whatever is chosen must be understood by both refgen and the conformance harness, and it changes what a spec body looks like — which is why this wants settling before the bulk of them are written. The existing suite’s checkSlicesLogInput already takes a per-call tolerance (§1.9), which is prior art for the numeric half; it says nothing about the “unspecified but deterministic per platform” half.

Commits. (1) Survey the known cases — DotProductPairs overflow, float conversion rounding, shuffle index out of range, saturation edges, NaN/±0 handling — classifying each as specified, implementation-defined, or undefined. (2) Choose and document the spec vocabulary in internal/spec/doc.go. (3) Implement it far enough that one operation with a genuinely implementation-defined corner round-trips through refgen and the harness.

Done when an operation with an architecture-dependent corner can be written in spec, generated into simdref, and conformance-tested without a false failure on any supported target.

Track B — Instrumentation and enforcement

Build specfill, then make it authoritative. Independent of tracks C, D and F; gen-docs gates three of track E.

specfill — API walker + report mode

Done: [x] · Needs: — · Blocks: fill-gen, doc-triage

The instrument the rest of the plan reads from.

specfill builds the whole of specdoc.Fill and cmd/specfill — the walk, the spec index, the comparison, and the rewrite of doc comments and parameter/result names. fill-gen wires it into generators with rewriting off; named-results and gen-docs then turn on name filling and doc injection there, and handwritten-fill turns -w on for hand-written files.

Interfaces. specdoc.Fill is the library entry point — Go source in, Go source out: func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error). Generators call it on their buffer before writing (§1.5). When verification checks fail, Fill returns a *Report as an error containing the findings enumerated below. Callers can recognize this error via errors.As if they choose to inspect or combine findings across files. Option fields:

• RejectUnknown controls whether an exported declaration with no spec entry is an error (recorded in Report.UnknownDecls). Off during the transition; generators flip it on after spec-all, so that adding an operation without writing its spec function fails at generation time rather than silently escaping coverage.
• AllowDocRewrite controls whether a declaration spec covers may already carry spec-owned paragraphs (§1.6). When false (default), each one that does is an error (recorded in Report.UnexpectedDocs); implementation notes and directives are always allowed. Once generators emit no API prose of their own (template-docs), this is their guard against silent doc loss (risk-silent-doc-loss): a generator paragraph missing its note prefix is an error rather than something Fill quietly replaces. Until then generators set it to true. Hand-written files carry prose by design, so cmd/specfill always sets it to true and relies on -diff and its -w guards instead.
• AllowNameMismatches controls whether name mismatches are tolerated. When false (default), they are errors. Generators can set it to true during the transition, until named-results reconciles them.

Two more fields select what Fill rewrites, for callers that want only part of it: NoFillDoc leaves doc comments alone, and NoFillNames leaves unnamed arguments unnamed (and so reports them as name mismatches). By default Fill does both. Finally, the Filename field labels positions in findings.

Names. Fill fills and checks names as §1.8 describes: an argument the declaration leaves unnamed (or names _) takes spec’s name, and a named argument whose name differs from spec’s is a mismatch. It rewrites a declaration’s names only when that declaration has no type or name mismatch, so a mismatch has to be fixed before the rest of its declaration’s names are filled.

cmd/specfill is a gentools-based command that applies specdoc to hand-written files: it walks the non-test .go files in simd and archsimd, skips generated ones (their generators call Fill themselves), calls Fill on each with AllowDocRewrite, and prints the combined report. -diff shows what a rewrite would change. With -w, rewrite in place; -w aborts without rewriting when the report has any findings, or if there are uncommitted changes, so its git diff is the complete story of what it did. The -no-fill-doc and -no-fill-names flags set the corresponding options. CI enforcement is a go test in this package — fill-enforce wires it — not a command invocation.

Report. The report is the permanent enforcement layer — every item in it remains useful after the transition as the API continues to evolve, and is what fill-enforce runs in CI. Coverage and name relations are transition metrics that converge to their targets; they stay in cmd/specstats / SPEC-STATS.txt and are not part of this report.

The report holds only errors; it does not preview rewrites. Every consumer of specdoc is a gentools program, cmd/specfill included, so -diff already shows exactly what a rewrite would change against the files on disk, including any text it would drop. That diff is how rewrites are reviewed, and a report that recomputed it would only be a second copy to keep in sync.

The possible errors are:

1. Type signature mismatches — parameter or result types, or their counts, disagree between spec and the declaration the walker found.
2. Name signature mismatches — a named parameter or result whose name differs from spec’s. Unnamed ones are filled rather than reported, unless NoFillNames is set. Omitted when AllowNameMismatches is set.
3. Check (a) violations — §1.6 ordering violations in declarations spec covers. Fill leaves such a comment as it is. Declarations spec does not cover are not checked: the ordering rule exists so rewrites are idempotent, and Fill never rewrites them. Hand-written files in particular may hold declarations spec will never cover. Each declaration is checked from the moment spec covers it.
4. Unknown declarations — exported declarations with no spec entry, when RejectUnknown is set.
5. Unexpected doc comments — declarations spec covers that carry spec-owned paragraphs, when AllowDocRewrite is false.

Commits. (1) specgen: add an index over Load()’s result keyed by (recv, name); add a function that classifies comment paragraphs as spec-owned or implementation note per §1.6; reject spec docs that break check (b). (2) New package _gen/specdoc and new command _gen/cmd/specfill: AST declaration walker, signature and comment ordering comparison, report findings implementing the error interface, and AllowDocRewrite / AllowNameMismatches options. Signature comparison translates the parsed *ast.FuncDecl into spec’s own representation — its types into specexpr types, which suffices because no qualified types exist (§1.4) — and compares that structurally against the specgen.Func. (3) Rewrite engine in specdoc.Fill (both doc comments and signature names) and in-place rewriting (-w, -no-fill-doc, -no-fill-names) in cmd/specfill.

Done when specfill, run on the current tree, writes nothing (without -w) and prints all report findings in position order.

SPEC-STATS.txt is the sanity check for signatures: cmd/specstats computes the same type and name mismatches, counting names the way Fill does (§1.8), so specfill’s report should agree with it line for line on those. Treat any disagreement as something to explain before shipping. Coverage and name relations stay in specstats as transition metrics; once they reach their targets, specstats can be retired.

fill-gen — Wire specfill into the generators

Done: [x] · Needs: specfill · Blocks: named-results, gen-docs

Each generator passes its buffer through specfill before writing, with all rewriting off for now: NoFillNames (named-results turns name filling on) and NoFillDoc (gen-docs turns doc injection on). Generators also set AllowNameMismatches, because names still disagree with spec until named-results, and AllowDocRewrite, because they still emit API prose of their own until template-docs.

So this task only wires the generators up, and generated output does not change. What does change is that Fill’s remaining checks now run at generation time: a type mismatch or a check (a) violation in generated output fails generation. Neither can be tolerated by an option, and as of specfill’s completion there are none in generated output.

Commits. (1) Add a narrow post-processing hook to gentools that does not make gentools depend on specgen (see §1.5). (2) Wire simdgen. (3) Wire tmplgen, wasmgen, midway.

Done when all four generators route through specfill, and generated output is byte-identical to the previous tree.

named-results — Reconcile parameter and result names with spec

Done: [x] · Needs: fill-gen · Blocks: fill-enforce

Every declaration spec covers carries spec’s parameter and result names. See §1.8. Declarations spec does not cover are out of scope; they gain names when spec-all covers them.

SPEC-STATS.txt §2 has the counts, split the way specfill treats them:

• Named arguments whose names disagree with spec, in generated or hand-written code. Each is fixed in whichever of spec or the API has the worse name — spec, a generator’s template or input, or a hand-written file; either way the diff touches only names, and doc text that mentions them. These come first: Fill fills no names in a declaration with a mismatch.
• Unnamed arguments, which Fill fills from spec. In generated code, that means turning name filling on in the generators. In hand-written files, specfill -w -no-fill-doc fills them; -w refuses while the report has findings, so this also needs any check (a) violations in hand-written files cleared (they are also doc-triage’s).

Cross-target parameter-name divergence (also SPEC-STATS.txt §2) is only fixed here for pairs spec covers. The rest reach zero as spec-all covers them, when each new spec function has to agree with every target at once.

Commits. (1) Reconcile name mismatches, grouped by generator or operation family, each commit with its regenerated output. (2) Stop setting NoFillNames and AllowNameMismatches in the generators, with the regenerated output: name filling is on, and a name mismatch in generated output fails generation. (3) Fill unnamed arguments in hand-written files with specfill -w -no-fill-doc.

Done when SPEC-STATS.txt §2’s name mismatches and unnamed counts are zero, and the generators run without NoFillNames or AllowNameMismatches.

gen-docs — Inject spec docs into generated output

Done: [x] · Needs: fill-gen · Prefers: named-results · Blocks: comments-yaml, handwritten-fill, template-docs · Risk: risk-silent-doc-loss

Turn on doc injection in the generators by dropping NoFillDoc. Generators keep setting AllowDocRewrite until template-docs removes their own prose.

Prefer landing this after named-results: spec’s docs refer to argument names (§1.8), so injecting them before the generators fill those names leaves docs that mention names the signature does not bind.

Expect a real diff, not a no-op: spec already overlaps several hundred (receiver, name) pairs (SPEC-STATS.txt §1), and almost none of the existing doc text matches spec’s (SPEC-STATS.txt §3). So this rewrites that many doc comments. Use go run . -diff first to size the change exactly.

Before any generator’s docs are injected, its implementation notes must follow §1.6, or the first rewrite silently drops them. Known gaps: some notes say CPU Features: or omit a colon; simdgen’s performance caveats (“A non-constant value of shift may result in significantly worse performance”, and the SetElem jump-table line) have no prefix, and the SetElem one sits directly above its Asm: line with no blank line between them. specfill’s check (a) violations and a -diff of the output with injection on are the list to work from.

Commits. (1) Fix note prefixes in simdgen, tmplgen, wasmgen and midway templates — Performance: for the caveats, a blank line before every note — with the regenerated output; the diff should touch only note lines. (2) Turn on doc injection in simdgen. (3) Turn it on in tmplgen, wasmgen, midway. Commits (2) and (3) include their regenerated output; verify that output changes only for operations spec covers, and review every changed doc comment to make sure nothing was unintentionally dropped. (4) If fill-enforce is already done, drop NoFillDoc: true from cmd/specfill.TestFill for generated files to start enforcing documentation matches.

Done when generated output has no check (a) violations, all four generators inject docs, nothing was dropped without review, and the only differences against the previous tree are doc comments for operations spec covers.

fill-enforce — Signature + name mismatch becomes an error

Done: [x] · Needs: named-results

For operations spec covers, disagreement fails. Missing spec functions remain tolerated. Note this does not depend on sig-refactor: specfill compares emitted source, not simdgen’s internal model.

Generated output is already checked as it is generated: type mismatches fail generation from fill-gen on, and name mismatches from named-results on. That misses hand-written files, and generated files edited by hand without rerunning their generator. This task closes both with a go test in cmd/specfill that runs Fill over every non-test .go file in simd and archsimd, generated or not, and fails on any finding. It sets AllowDocRewrite, since those files carry spec-owned docs by design, whether injected or hand-written.

Commits. One, adding the test.

Done when a deliberately introduced signature or argument-name change in any generated or hand-written exported declaration fails the build.

Track C — Conformance

Behavioral testing against simdref. Parallel to everything else; shares only the spec-coverage bottleneck.

ci-audit — CI/hardware audit

Done: [x] · Needs: — · Blocks: conform-fixed · Risk: risk-conformance-env

Determine what can actually execute tests for amd64, arm64, wasm, and SVE — natively or emulated. The conformance suite’s value for other architectures depends on it.

“Execute tests” means: build simd and archsimd for the target and run go test against real or emulated hardware, at a speed that tolerates a few thousand generated conformance cases. SVE needs SVE-capable hardware or QEMU, and wasm needs a runtime.

Done when one document or issue records, per target, whether it can run today, on what, how fast, and what it would take if not — marking each runnable, runnable-with-work (with the work named), or not runnable, so conform-fixed knows which targets it is writing tests for.

Results

For amd64, use the gotip-linux-amd64_avx512 gomote for testing.

qemu-aarch64 -cpu neoverse-n1 (which matches the capabilities of gotip-linux-arm64) is roughly 7x slower than gotip-linux-arm64, so well within the range of practical conformance testing.

For NEON, use the gotip-linux-arm64 gomote for testing. For SVE, use qemu-aarch64 -cpu neoverse-v1.

node is about 3x faster than wazero.

For wasm, use node for testing.

conform-fixed — Conformance harness, fixed-width types

Done: [ ] · Needs: ci-audit · Blocks: validate-amd64, conform-scalable · Prefers: specfill

Generated tests comparing archsimd against internal/simdref. The generator’s input is spec (hand-written source), so this is not a generator reading another generator’s output.

Commits. (1) Input generation: start from archsimd/internal/simd_test/helpers_test.go, which already has the per-type edge-case vectors, the mask patterns and the window-walking helpers (§1.9); decide whether to lift them into a package the generated tests can import. (2) Test generator producing one test per (recv, name) spec covers. (3) Wire into the SIMD test suite, skipping operations whose spec body panics.

Done when every operation spec covers, on every target ci-audit marked runnable, is exercised against simdref with both edge-case and random inputs, and failures name the operation, the input, and both results.

conform-scalable — Conformance harness, scalable types

Done: [ ] · Needs: width, conform-fixed · Blocks: conform-flip

Extend conform-fixed’s harness to the scalable types (Int8s … Float64s, Mask8s … Mask64s) on SVE. Separate from conform-fixed because it cannot be written until width settles how spec executes at a concrete width: the harness must construct simdref values at the width the hardware reports through vl() rather than at spec’s fixed scalableWidth.

Commits. (1) Teach input generation to size vectors from the runtime vector length. (2) Extend the test generator to scalable receivers. (3) Wire into the suite, gated on SVE availability per ci-audit.

Done when scalable operations spec covers are conformance-tested on SVE at the hardware’s actual vector length.

validate-amd64 — Validate spec bodies against amd64

Done: [ ] · Needs: conform-fixed, spec-common, impl-defined · Blocks: conform-flip

Run conformance with amd64 as the oracle. Every failure is a spec bug until proven otherwise. See §1.9.

Commits. One per batch of spec fixes; keep spec changes separate from harness changes so the reason for each is legible.

Done when the harness passes on amd64 for every operation spec covers, with every remaining difference either fixed in spec or recorded as implementation-defined under impl-defined’s policy.

conform-flip — Flip conformance direction

Done: [ ] · Needs: validate-amd64, conform-scalable

Spec becomes the oracle for every target, not just amd64. The second half is the contributor-facing note — likely in internal/spec/doc.go and the SIMD README — describing the workflow: add the operation to spec if it is not there, implement it, run conformance.

Done when conformance runs and fails the build on all targets ci-audit marked runnable, and the workflow is written down where someone adding an architecture will find it.

Track D — Spec content

Writing spec functions, docs, and bodies. The shared bottleneck the other tracks draw on.

doc-explore — Doc-mechanism exploration on a hard family

Done: [x] · Needs: — · Blocks: spec-common · Risk: risk-doc-template-ceiling

As of CL 831485, roughly one doc line in eight in categories.yaml carries shape-specific literals — [x0, x1, x2, x3, ...], 128-bit, 0b_11_01_00_10 — concentrated in AddSub’s Concat*Pairs and Moves' concatSelectedConstant*. Those docs cannot be produced by substituting {vL} into one template; each shape needs its own text.

The expected mechanism is multiple spec functions sharing one //specgen:name, which specgen supports because funcKey is (recv, name) and the constraint sets are disjoint. Verify that before bulk migration, not after: discovering it does not scale once 150 easy docs are written means retrofitting all of them.

Commits. (1) Migrate ConcatAddPairs / ConcatSubPairs into spec, docs included; their text differs between 16/32-bit and 64-bit elements. Verify via cmd/specls. (2) Migrate the Grouped variants, whose docs vary by element size and by the 128-bit grouping — two axes, and the real test of the mechanism. (3) ShiftSaturated, ShiftAllRight, ShiftRight, whose docs differ between signed and unsigned elements; categories.yaml selects the variant with a signed: join key, which this retires. (4) If the mechanism proves insufficient, extend it (conditional doc text is the likely shape) and document the extension in internal/spec/doc.go.

concatSelectedConstant* is harder still — four spec functions for one API name — but it is unexported, so it sits outside specfill’s scope per §1.7 and is undecided until compiler-ops. Reach for it only if compiler-ops has landed and you want a sterner test.

Done when cmd/specls reproduces the existing doc text for all three families, or the doc mechanism has been extended so that it can.

spec-common — Spec the common operations

Done: [x] · Needs: width, doc-explore · Blocks: validate-amd64, comments-yaml, spec-all · Prefers: specfill · Risk: risk-ref-impl-effort

The method set midway/comments.yaml documents: its .common_methods block less String and Len — Add, Sub, Min, Max, And, Or, MulAdd, the compares, and the rest of the portable core. SPEC-STATS.txt tracks it as “portable core”, which is this task’s completion bar. Highest leverage in the plan — these are the easiest reference implementations to write correctly, they unlock comments-yaml (the worst doc divergence, in the widest-audience package), and they are exactly what a new architecture implements first.

The file’s keys are not all operations: besides the method docs, it documents vector types, functions and templates, and carries receiver names keyed to the shared block. Only the methods/functions are in scope here, and this excludes non-spec residue (String, etc, §1.7).

Commits. Start by sampling ~10 operations at random and timing how long it takes to implement them, to calibrate risk-ref-impl-effort before committing to the rest. Then one commit per family (arithmetic, bitwise, min/max, compares, multiply-accumulate), each adding the spec functions with docs and bodies and confirming cmd/specls output.

Done when every method name in midway/comments.yaml other than non-spec residue resolves to a spec function, with a real body, unblocking comments-yaml.

spec-all — Bulk spec migration

Done: [ ] · Needs: spec-common, convert-lo, mask-bits · Prefers: specfill, fill-enforce · Blocks: template-docs · Risk: risk-ref-impl-effort

The remaining exported operations, prioritized by the names produced by more than one generator (SPEC-STATS.txt §1) — highest drift risk, and each migration fixes several files at once. Best-effort bodies are acceptable; panic("not implemented") is safe because conformance skips it. A wrong body is worse than no body.

Two denominators, both useful. Against the API: the (receiver, name) pairs SPEC-STATS.txt §1 shows uncovered, which is what the scoreboard tracks. Against the input being retired: categories.yaml’s exported operation names, less the ones spec-common handles. They are different counts of different things — one categories.yaml name can expand to many API names — so quote whichever the audience needs and say which.

Why convert-lo and mask-bits gate this. Only the Converts and LoadStore families actually depend on them — convert-lo may rename ExtendLo* / ConvertLo*, and mask-bits may rename the scalable mask bitmap pair, and renaming after the docs are migrated means doing them twice. If either is still open when this task starts, the other families are unaffected and are the place to begin.

Commits. One per operation family, mirroring the ops/*/ grouping. Each spec function gains //specgen:commutative where the operation is commutative; the directive already exists and is parsed by specgen. Drop any comments.yaml entries spec gains as they are covered. Include all regenerated outputs in each commit. Once gen-docs has landed, each family’s regeneration rewrites that family’s docs, so review the diff to ensure all doc comment content is equivalent: any dropped text is either something spec should say or an implementation note missing its prefix.

Done when specfill reports no remaining API declarations uncovered by spec, other than the permanent residue named in §1.7 (String, Len).

Track E — Retiring the old inputs

Retiring the other documentation sources, then categories.yaml itself.

doc-triage — Hand-written doc conflict triage

Done: [ ] · Needs: specfill · Prefers: spec-common · Blocks: handwritten-fill · Risk: risk-silent-doc-loss

Make it safe for handwritten-fill to rewrite hand-written files, by emptying the conflict set first.

A conflict is a declaration where spec has a doc and the hand-written file has different spec-owned paragraphs (§1.6). Those are the only sites -w would overwrite: where spec has no entry, the declaration is left alone. Every conflict therefore resolves one of two ways — the prose says something spec should say (move it into spec), or it is about how a target implements the operation (make it an implementation note by giving it one of §1.6’s prefixes, and specfill never touches it). Neither outcome loses text, which is what makes -w safe afterwards.

Run specfill -diff over the hand-written files — every doc comment it would rewrite is a conflict — and sort each conflict into one of those two. SPEC-STATS.txt §3 breaks the conflict count down by file, with simd_emulated.go dominating and a long tail of archsimd/slicepart_*.go, shuffles_amd64.go, and extra_amd64.go. Some implementation text there has no prefix today — extra_amd64.go’s “This method compiles to VPTEST x, x” is a Performance: note — and the same pass fixes check (a) violations and note spellings in these files.

The slicepart_*.go files matter most: they declare the Load*Part family spec already covers via LoadZPart, so their prose conflicts on day one — and one of them documents LoadUint32x4Part as loading “a Int32x4”.

Done when specfill -no-fill-names -diff is empty and specfill reports zero check (a) violations.

comments-yaml — Drop spec-covered docs from midway/comments.yaml

Done: [x] · Needs: gen-docs, spec-common · Prefers: fill-enforce

Every method or function-template entry spec covers is dropped from comments.yaml. At the time of this task (after spec-common) that is the portable core less non-spec residue (String, etc, §1.7), plus the Load/LoadPart/Broadcast templates. Entries spec gains afterwards (in spec-all) are dropped when spec gains them.

Commits. (1) Confirm spec covers the portable core; fill gaps. (2) Delete the covered entries from comments.yaml. (3) Run specfill -w over simd/simd_emulated.go — this overlaps handwritten-fill; sequence whichever lands first.

Done when comments.yaml has no entry for any operation spec covers, midway sources those docs through specfill, and simd_stubs.go and simd_emulated.go carry identical spec-derived docs for them.

handwritten-fill — Enable -w on hand-written files

Done: [ ] · Needs: gen-docs, doc-triage · Risk: risk-silent-doc-loss

Turn on doc rewriting for hand-written files, once doc-triage has emptied the conflict set — at which point -w rewrites only docs that already match what it would write.

Two guards, both on cmd/specfill only. This does not change gentools' shared -w, which every generator uses:

• specfill -w refuses to run when the report has any findings, or if there are uncommitted changes, so its diff is the complete story of what it did.
• specfill -w never runs in CI; CI runs report mode (fill-enforce).

Done when specfill -w over the hand-written files is a no-op on a clean tree.

template-docs — Retire generator-supplied API docs

Done: [ ] · Needs: gen-docs, spec-all · Blocks: cat-rehome

Delete the doc strings embedded in wasmgen’s and tmplgen’s templates, and the documentation: fields on exported categories.yaml entries, leaving specfill to supply them. Mechanical once spec-all has covered the operations those generators emit — but not automatic, because their templates interleave prose with the implementation notes that stay with the generator (§1.6).

Commits. One per generator, stripping the prose and keeping the notes; one for the documentation: fields on exported categories.yaml entries. Each includes its regenerated output. Then stop setting AllowDocRewrite in the generators, so a generator that emits API prose again fails generation.

Done when no generator’s source contains API prose, no exported categories.yaml entry has documentation:, regeneration is byte-identical to the previous tree, and the generators run without AllowDocRewrite.

cat-rehome — Re-home categories.yaml non-doc fields

Done: [ ] · Needs: template-docs · Blocks: compiler-ops-impl, sig-from-spec · Risk: risk-cat-unknowns

Empty the exported entries of categories.yaml of everything but go: and commutative:, by sending each remaining field where §1.3’s inventory says it belongs. template-docs has already removed documentation:; this is the rest. Compiler-only entries are untouched, for compiler-ops-impl.

Expect surprises. §1.3’s audit was static, and the unifier’s behavior on field combinations was not exhaustively traced — regexpTag’s role is still unverified (§1.3). Mitigating factor: go run . -diff makes every step verifiable byte-for-byte against checked-in generated output.

Commits. Follow §1.3’s disposition column, one field per commit, confirming generated output is unchanged at each step: (1) delete regexpTag and signed from categories.yaml and from go_*.yaml, expecting most go_*.yaml uses to be inert already; (2) move constImm into go_amd64.yaml; (3) convert the out/elemBits/bits entries into spec function splits, which spec-all should already have done — verify.

Done when every exported entry in categories.yaml has only go: and commutative: keys; compiler-only entries are untouched, for compiler-ops-impl. Generated output is byte-identical throughout.

compiler-ops-impl — Implement compiler-only ops representation

Done: [ ] · Needs: compiler-ops, cat-rehome · Blocks: sig-from-spec

Implement whatever compiler-ops decided, for the compiler-only entries — those that produce no exported declaration — and the noTypes / noGenericOps switches. No design work should remain at this point; if it does, compiler-ops was not finished.

These operations have no API surface — they exist only as SSA ops and lowering rules — so nothing here touches specfill, spec, or the public packages. It is entirely about giving simdgen somewhere other than categories.yaml to read them from.

Done when no compiler-only entries remain in categories.yaml (SPEC-STATS.txt §4) and generated output is byte-identical.

cat-delete — Delete categories.yaml

Done: [ ] · Needs: sig-from-spec

Delete ops/*/categories.yaml, the !import stub that loads it, and its handling in simdgen/main.go. By this point every field it carried has been re-homed (cat-rehome), its compiler-only entries have moved (compiler-ops-impl), and its signatures come from spec (sig-from-spec), so this should be pure deletion.

Done when categories.yaml no longer exists, simdgen runs without it, and generated output is byte-identical.

Track F — simdgen internals

The signature refactor and, eventually, sourcing signatures from spec.

sig-refactor — simdgen signature refactor

Done: [ ] · Needs: — · Blocks: sig-from-spec · Risk: risk-sig-refactor-convergence

Make the Go signature a first-class value on Operation instead of an emergent property of shape() → classifyOp() → a template name. Ideally shaped so it is directly comparable to specgen.Func’s Recv/In/Out.

This collapses the stub templates (§1.1 counts them) to roughly three, because operand permutation and parameter naming become data rather than template identity; the _21, _231Type1, _31 suffixes become an explicit permutation.

It is also the empirical test of whether the unification approach can be kept. The refactor assumes the signature is expressible as data without a new pile of special cases; the quirk fields and simdgen only supports… panics catalogued in §1.1 are mostly upstream of the signature (XED inconsistency) or downstream (register shapes, rules), so the seam looks thin — but that is a prior, not a measurement. If it does not converge, that is the evidence for revisiting §1.1, earned cheaply on the few hundred lines §1.1’s layer table isolates.

Commits. (1) Introduce the signature type and a GoSignature() method covering the simplest classes (op1, op2, op3). (2) Extend to immediate variants (op*Imm*), which may be fiddly. (3) Extend to permuted and VecAsScalar variants. (4) Rewrite the stub templates to consume the signature value; delete the redundant templates.

Done when go run . -diff shows byte-identical output at every commit.

sig-from-spec — simdgen sources signatures from spec

Done: [ ] · Needs: sig-refactor, cat-rehome, compiler-ops-impl · Blocks: cat-delete

simdgen looks up the spec function by (recv, name) and asserts signature equality, reporting a structured diff on mismatch. Key the lookup on (recv, name) only — spec’s own uniqueness key — and assert the signature separately. Keying on the signature makes a divergence indistinguishable from “not migrated yet”, which is the failure this whole effort exists to catch.

This is where sig-refactor pays off: the lookup compares against the signature value that refactor introduced, rather than against a rendered template.

Commits. (1) Look up the spec function and assert equality, failing the generator run on mismatch. (2) Delete whatever signature information in go_GOARCH.yaml is now redundant. (3) Switch simdgen to reading commutativity from spec’s //specgen:commutative directive and delete commutative: from exported categories.yaml entries.

Done when simdgen derives every exported signature it emits from spec, commutativity comes from spec, and generated output is byte-identical.

---

Part 4 — Risks and unknowns

Every risk is owned by a task, and its substance is stated there. This is the index. risk-concrete-width, risk-impl-defined-behavior and risk-ref-impl-effort are the expensive ones: each reaches every spec body, so each is far cheaper to settle before the bodies are written than after.

---

Appendix A — Regenerating the two generated files

SPEC-STATS.txt

cmd/specstats generates SPEC-STATS.txt, the transition scoreboard:

cd $GOROOT/src/simd/archsimd/_gen
go run . -w                   # (if necessary) ensure generator outputs are up-to-date
go run ./cmd/specstats        # print the report
go run ./cmd/specstats -diff  # see what's changed
go run ./cmd/specstats -w     # rewrite SPEC-STATS.txt in place

It loads simd and archsimd through go/packages for each build configuration, taking syntax only — no type-checking, per §1.4 — and reads simd/internal/spec through specgen.Load. It needs a recent Go toolchain and up-to-date output from the SIMD generators, but does not itself require any ISA data. Commit the regenerated file alongside whatever change moved the numbers; the diff is the record of what moved.

It reports only figures that measure how far along the transition is: every one is a fraction heading for 100% or a count heading for zero, tagged with the tasks that move it; indented, untagged numbers are context breaking down the figure above them. If you add a figure, give it a denominator and a direction, or leave it out.

One thing it deliberately does not do: it classifies divergence (§5) but fails on nothing; enforcement is specfill’s job. specfill should absorb the comparison half of this program rather than leave two implementations to drift.

For one-off questions the report does not answer, cmd/specls prints the API spec currently expands to:

go run ./cmd/specls              # the whole expansion
go run ./cmd/specls -f Add       # one spec function
go run ./cmd/specls -f Add -trace  # why it expands the way it does

SPEC-TASKS.dot

The task graph is drawn from Part 2’s two tables by cmd/spectasks:

go run ./cmd/spectasks        # print the graph
go run ./cmd/spectasks -w     # rewrite SPEC-TASKS.dot in place
dot -Tsvg SPEC-TASKS.dot -o /tmp/tasks.svg  # render to SVG
go run ./cmd/spectasks | grep -v ^-- | xdot -  # view the current graph

It reads nothing but this file. Two things it does beyond drawing:

It checks, from go test ./cmd/spectasks. Dependencies are stated three times here — Part 2’s table, Part 3’s Needs:/Prefers: lines, and Part 4’s Blocks: lines, which are the same edges reversed — and the test fails if the three disagree, if the table is not topologically sorted, if a task is in no track or in two, or if an edge points at a task that does not exist. A second test fails if SPEC-TASKS.dot no longer matches the document, so an edit that skips the regeneration step is caught too. The command itself refuses to write a graph from an inconsistent document, but the test is where the problems are reported.

It derives the critical path rather than restating one: a task is on it when its longest chain in plus its longest chain out spans the whole graph. That keeps every longest chain, not one picked by hand, so parallel roots of equal depth are all marked.