Go: A Documentary

Diagnostics and knobs

Part I · 21 threads in 2 era slices, from Go 1.0 to Go 1.28. Each thread gives its design, then a per-release timeline of what landed (Done) and what was still open. Release labels in a timeline link to that release's page.

● = the thread had substantive commits in that cycle; hover for the stage summary, click to jump to it in the release page.

Thread map, Go 1.0 – 1.17

Thread1.01.11.21.31.41.51.61.71.81.91.101.111.121.131.141.151.161.17Status at go1.17 (or at the end of its era slice)
Diagnostics and knobs (Go 1.0–1.17)
Execution tracer (trace v1): events, per-goroutine sequencing, annotations●●●●●●●●still the v1 design at go1.17 (per-P buffers, text of events decoded by internal/trace, go tool trace UI) → continues as The new execution tracer (trace v2): generations, per-M buffers, new parser (1.18+)
Race detector integration●●●●●●●●●●●●●●●●still the compiler-instrumentation (racewalk) + TSan-runtime design at go1.17; ports on linux/amd64, darwin, freebsd, windows, netbsd, openbsd/amd64, ppc64le, arm64, darwin/arm64 → continues as Race detector, ASan, MSan and Valgrind integration (1.18+)
CPU profiler and runtime/pprof writers (legacy format to compressed protobuf)●●●●●●●●●●●●●●●●●still the design at go1.17: signal-driven SIGPROF samples logged to a profile buffer, aggregated by runtime/pprof goroutines and streamed as gzip protobuf with inline-aware Locations → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+)
Profile labels●●●●●still the design at go1.17: g.labels pointer, copied by the signal handler into the CPU profile tag buffer → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+)
Heap profile accounting and MemStats●●●●●●●●●still the design at go1.17 (consistent stats partly sharded; MemStats copied explicitly), with heapGoal in gcControllerState → continues as runtime/metrics and MemStats (1.18+)
runtime/metrics●●still the design at go1.17 → continues as runtime/metrics and MemStats (1.18+)
GODEBUG, runtime/debug, debug log and traceback knobs●●●●●●●●●●●●●●●still the design at go1.17 → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)
MSan and sanitizer integration●●●●●msan (linux/amd64, arm64) in place; ASan not yet present → continues as Race detector, ASan, MSan and Valgrind integration (1.18+)
GOEXPERIMENT handling (objabi, buildcfg, goexperiment)●regabi sub-experiments on by default in the baseline for amd64 (and Android) → continues in internal/buildcfg/internal/goexperiment (1.18+, see the later report's experiment threads)
Traceback correctness and inlining (diagnostic output)●●●●●●●still the design at go1.17 (gentraceback) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)
GDB runtime support●●●●●●●runtime-gdb.py maintained (Python 2/3, gdb 7.12–9.2) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)

Thread map, Go 1.18 – 1.28

Thread1.181.191.201.211.221.231.241.251.261.271.28Status at HEAD
Runtime: tracing, profiling, metrics and diagnostics (Go 1.18–1.28)
The new execution tracer (trace v2): generations, per-M buffers, new parser●●●●●●●●●●●default since 1.22 (experiment exectracer2 removed, v1 deleted in 1.23); trace format versions 1.22 → 1.23 → 1.25 → 1.26; flight recorder shipped in 1.25; ongoing hardening and perf work
Frame-pointer unwinding in tracing and profiling; profile stack depth●●●●●FP unwinding used by tracer, block and mutex profilers; stack depth 128 default (GODEBUG=profstackdepth); tracefpunwindoff documents which profilers it governs
Mutex and block profiles, including runtime-internal lock contention●●●●●●●runtime-internal lock contention reported in the mutex profile with sync.Mutex-like semantics (blame the unlocker); runtimecontentionstacks setting removed
Flight recorder and trace subscribers●●shipped in runtime/trace (1.25); in-band generation-end signal added 1.26
CPU profiling, goroutine profiling, pprof labels and the protobuf writer●●●●●●●●●●●maintenance; labels in tracebacks behind GODEBUG=tracebacklabels=1
Goroutine leak profile (GC-based leak detection)●●goroutineleak profile on by default; GOEXPERIMENT=goroutineleakprofile removed
runtime/metrics and MemStats●●●●●●●●~dozens of metrics; Read guaranteed not to stop the world (test)
Coverage redesign: runtime/coverage, internal/coverage, covdata●●●●●●default since 1.20; coverageredesign experiment removed (41298239cf, Jan 2025)
Race detector, ASan, MSan and Valgrind integration●●●●●●●●●●●race on linux/riscv64, loong64, s390x; asan/msan linux incl. loong64, riscv64, ppc64le; build-tag Valgrind support
Crash output, runtime/debug, BuildInfo and debuglog●●●●●●●debug.SetCrashOutput public; build info includes VCS and settings; debuglog compiled in tests

Diagnostics and knobs (Go 1.0–1.17)

Reader: Claude Sonnet. Cites 528 of 915 commits in this slice; every non-noise commit is cited, plus 5 larger commits the noise filter had set aside, which a separate gap reader described. Read the design-heavy messages in full and the first lines of the rest. The mutex-profile introduction falls in the scheduler bucket and is covered there.

Release digest for this slice
  • Go 1.0 — Everything up to Go 1: runtime/pprof CPU and heap profiling, runtime/debug, pprof.Profile, GOMAXPROCS/NumCPU, gdb support, in the profiler and pprof threads.
  • Go 1.1 — Race detector added (compiler racewalk, TSan runtime, Windows port, lazy shadow); runtime/debug GC controls; heap profile overhead cut from 64kB to 2kB per MB.
  • Go 1.2 — Block profiler gains select/close support; race instrumentation fixes; “created by” in tracebacks.
  • Go 1.3 — Race calls via assembly thunks rather than cgo (about 40-50% faster); profiling and Goexit fixes.
  • Go 1.4 — Runtime converted from C to Go reaches profiling (cpuprof, MemProfile, traceback) and race code.
  • Go 1.5 — Execution tracer, go tool trace and runtime/trace introduced; sequence-numbered ordering; GODEBUG gccheckmark and memprofilerate.
  • Go 1.6 — msan runtime support; tracer made free of write barriers; debug.SetTraceback; panic frames restored in tracebacks.
  • Go 1.7 — Per-goroutine sequence numbers cut tracing overhead (453% → 26% at 48 cores); file:line in traces; per-P TSan contexts.
  • Go 1.8 — pprof protobuf output for CPU, heap, goroutine and counted profiles; GC events in traces; MemStats documented and NextGC made the real heap goal.
  • Go 1.9 — Streaming protobuf writer, profile labels, inlining-aware tracebacks/Callers, concurrent runtime.GC(), heap-profile snapshot at mark termination.
  • Go 1.10 — Block/mutex profiles in new format with delays in ns and mutex scaling; STW trace events; label-race fix.
  • Go 1.11 — Trace user annotations (Task/Span/Log), allocs profile, race on ppc64le, pprof mapping flags.
  • Go 1.12 — Race on arm64, debug.ReadBuildInfo, GODEBUG=madvdontneed, mid-stack inlining traceback work, gctrace=2 removed.
  • Go 1.13 — Runtime dlog ring-buffer debug log; go env -w/-u; arm/mips CPU profile corruption fixed.
  • Go 1.14 — Inlined function encoding in profiles, asyncpreemptoff GODEBUG, race support for new timers.
  • Go 1.15 — Max RSS in heap profiles, goroutine profile labels, block/mutex seconds parameter, race .syso rebuilds.
  • Go 1.16 — runtime/metrics introduced with histograms; consistent memstats; default to MADV_DONTNEED; race on darwin/arm64; msan memmove.
  • Go 1.17 — More metrics (scheduling latency, allocs/frees); block profile bias fix; mem profiling disabled when unreachable; race on openbsd/amd64 and register ABI for race calls; GOEXPERIMENT refactor with regabi sub-experiments on by default and experiments in runtime.Version().

Execution tracer (trace v1): events, per-goroutine sequencing, annotations

Span: Go 1.5 → Go 1.11 · Status at go1.17: still the v1 design at go1.17 (per-P buffers, text of events decoded by internal/trace, go tool trace UI) → continues as The new execution tracer (trace v2): generations, per-M buffers, new parser (1.18+)

Design docs & discussion

Design. The tracer was added in one stack of CLs by Dmitry Vyukov from a design document: a runtime part (traceEvent, per-P traceBuf buffers that are flushed as “batches”, a stack table hashed by PC sequence, a string table), runtime/pprof entry points StartTrace/StopTrace, a trace parser in internal/trace, and the go tool trace command. Events are compact varint-encoded records with delta timestamps; a batch event announces the P and base timestamp, and the parser merges batches and restores a consistent order. Stacks and (later) strings are interned into tables that are written at the end of the trace.

Event ordering was the hard problem. In Go 1.5 the first approach was timestamps (cputicks); because of unsynchronised clocks, VMs, non-x86 timer resolution and identical timestamps this produced flaky tests, so the ordering was made explicit with a global event sequence number (80c98fa901) while timestamps were kept for time display. A global sequencer slowed parallel programs by up to 10x, so in Go 1.7 it was replaced by per-goroutine sequence numbers assigned only to three event types (goroutine start, unblock, syscall exit); the parser rebuilds a partial order (a3703618ea: 5026 ns/op without tracing, 27803 with the global sequencer, 6369 with the new scheme at GOMAXPROCS=48; average event size 4.63 → 3.62 bytes). Go 1.7 also embedded file:line info for stack frames into the trace so it no longer needs the binary (0fb7b4cccd), with a version in the trace header that lets the parser still read 1.5 traces (3fafe2e888).

From Go 1.8 the tracer grew GC visibility (mark worker types, mark assists, STW events, sweep statistics) and in Go 1.11 a user annotation API (Task, Region/Span, Log) that bumped the trace version to 1.11 and required locking the string table.

Timeline.

Race detector integration

Span: Go 1.1 → Go 1.17 · Status at go1.17: still the compiler-instrumentation (racewalk) + TSan-runtime design at go1.17; ports on linux/amd64, darwin, freebsd, windows, netbsd, openbsd/amd64, ppc64le, arm64, darwin/arm64 → continues as Race detector, ASan, MSan and Valgrind integration (1.18+)

Design. The race detector (Dmitriy Vyukov) is the ThreadSanitizer (TSan) runtime linked as .syso files in runtime/race, driven by compiler instrumentation. Under -race cmd/gc/racewalk.c rewrites the AST so that every memory access calls runtime.raceread/racewrite (or the range variants for aggregates), function entry/exit call racefuncenter/racefuncexit, and the runtime calls acquire/release annotations from channels, locks, sync/atomic, goroutine creation and finalizers. Instrumentation is skipped for non-escaping, blank, and zero-sized values and for runtime-internal helpers.

Throughout 1.1 the compiler pass was hardened case by case (nested structs, selectors, arrays, &&/||, for conditions, type switches, append, OKEY, OEFACE), and its cost was reduced: racefuncenter takes its PC directly (ce287933d6, about 46-48% faster on encoding/json), and in 1.3 the cgo-based calls were replaced by custom assembly thunks that extract the caller PC, check that the address is heap or global and switch to the g0 stack (a1695d2ea3: regexp 3.69s → 2.23s, bzip2 9.46s → 4.70s, json 6.38s → 3.63s). In 1.4–1.5 the runtime was converted to Go, so the race detector had to ignore accesses on g0/gsignal stacks (467a6d28be) and race.c became Go (59e3e5354d). In 1.7 the TSan runtime was given per-P contexts, which fixed unbounded growth of sync object descriptors (caa2147532; an alternate procwire scheme was rejected as too messy).

Timeline.

CPU profiler and runtime/pprof writers (legacy format to compressed protobuf)

Span: Go 1.0 → Go 1.15 · Status at go1.17: still the design at go1.17: signal-driven SIGPROF samples logged to a profile buffer, aggregated by runtime/pprof goroutines and streamed as gzip protobuf with inline-aware Locations → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+)

Paths not taken: Background CPU profiling API (never landed)

Design docs & discussion

Design. CPU profiling began (2011) as a hash table in the signal handler (cpuprof) with an eviction log and a legacy binary format read by the pprof tool. Go 1.4 converted cpuprof from C to Go (ac49e6735b) and 1.5 simplified it (3c8a89daf3, 481ce36ec4). The profile format was replaced in Go 1.8: first conversion code from legacy to protobuf (7d14401bcb, f72a629dbd), then CPU (76f12cdaa2), count profiles (ccd69d0582), heap (86ab09eed5) and goroutine locations (c6228ef7e2) were emitted as gzip-compressed pprof protobuf at debug=0. The first implementation buffered the profile three times (in-memory profile, struct representation, marshalled bytes); Go 1.9 added a streaming protobuf encoder so only one copy exists (cbab65fdfa) and switched CPU profiling to a plain sample log (“new profile buffers”, 1a680a902a), with hash-based dedup done in an ordinary goroutine, a more efficient staging hash table (1564817d8c) and symbolization (cbef450df7). Inlining-aware encoding followed (Go 1.9: 9e83c11fca; Go 1.14 corrected inlined functions into one Location with multiple Lines, e038c7e418, e25de44ef2, 71c59ee6fc, and later fadbf7404d expanded the final truncated frame so locations are not split).

Timeline.

Profile labels

Span: Go 1.9 → Go 1.15 · Status at go1.17: still the design at go1.17: g.labels pointer, copied by the signal handler into the CPU profile tag buffer → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+)

Design docs & discussion

Design. Labels (pprof.WithLabels, Labels, Label, ForLabels, Do, SetGoroutineLabels) attach key/value pairs to goroutines. The runtime stores an opaque pointer labels in g, inherited at goroutine creation, set via runtime_setProfLabel and read by the SIGPROF handler into the CPU profile buffer; the proto writer then propagates them into samples. Because the signal handler cannot use real acquire/release, a global labelSync models happens-before for the race detector.

Timeline.

Heap profile accounting and MemStats

Span: Go 1.0 → Go 1.17 · Status at go1.17: still the design at go1.17 (consistent stats partly sharded; MemStats copied explicitly), with heapGoal in gcControllerState → continues as runtime/metrics and MemStats (1.18+)

Design. MemStats is a snapshot assembled by updatememstats stopping the world. The heap profile is built from mProf buckets attached to sampled allocations by “specials” on spans; sampling is exponential with MemProfileRate. In Go 1.9 the profile was made cycle-consistent: a global heap profile cycle counter is incremented during mark termination (cheap snapshot) and cleaned later (eee85fc5a1, 3ebe7d7d11, diagrammed in 673a8fdfe6) so a free cannot be attributed to an earlier cycle than its alloc. In Go 1.16 next_gc, stacks_inuse, gcWorkBufInUse etc. were turned into consistent per-P sharded stats (f77a9025f1, ae585ee52c) in preparation for runtime/metrics; the heap goal then moved into gcControllerState (3eaf75c13a).

Timeline.

runtime/metrics

Span: Go 1.16 → Go 1.17 · Status at go1.17: still the design at go1.17 → continues as runtime/metrics and MemStats (1.18+)

Design docs & discussion

Design. runtime/metrics exposes named metrics (/gc/heap/allocs:bytes style, with unit suffix) via All() descriptions and Read([]Sample), where the value is a Value union (uint64, float64, *Float64Histogram). Implementation in runtime/metrics.go computes values from sharded consistent stats and the runtime’s timeHistogram, a concurrent HDR-style histogram with log-linear buckets (36c5edd8d9). Histogram semantics were changed before release: shifted buckets to allow negative durations (b116404444), explicit boundaries rather than implicit infinite buckets (ae97717133), units aligned with bucket unit (32afcc9436), marked cumulative (6250833911).

Timeline.

GODEBUG, runtime/debug, debug log and traceback knobs

Span: Go 1.0 → Go 1.17 · Status at go1.17: still the design at go1.17 → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)

Design. GODEBUG is parsed by the runtime into a table of dbgvar entries (name, address); GOTRACEBACK was extended and made settable from code. Each transition below is a separate commit.

Timeline.

MSan and sanitizer integration

Span: Go 1.6 → Go 1.16 · Status at go1.17: msan (linux/amd64, arm64) in place; ASan not yet present → continues as Race detector, ASan, MSan and Valgrind integration (1.18+)

Design. -msan makes the compiler insert msanread/msanwrite and the runtime wrap allocations with msanmalloc/msanfree, calling the C MSan runtime via runtime/msan through cgo-free thunks. Instrumentation is skipped on the system stack and when g or m is nil (early init).

Timeline.

GOEXPERIMENT handling (objabi, buildcfg, goexperiment)

Span: Go 1.17 → Go 1.17 · Status at go1.17: regabi sub-experiments on by default in the baseline for amd64 (and Android) → continues in internal/buildcfg/internal/goexperiment (1.18+, see the later report’s experiment threads)

Design. GOEXPERIMENT is a comma-separated list; the parser lives in internal/buildcfg with generated boolean and integer constants in internal/goexperiment (b675e52e95 consolidates them, e.g. goexperiment.RegabiWrappersInt). The baseline set depends on GOOS/GOARCH so it moved out of goexperiment into objabi (cf2396c70e) and then buildcfg. Regabi is an alias that turns all sub-experiments on, and noregabi all off (14dbd6e776), not a flag in itself. The set of non-default experiments is recorded in runtime.Version() and go version (5159c83641).

Timeline.

Traceback correctness and inlining (diagnostic output)

Span: Go 1.0 → Go 1.17 · Status at go1.17: still the design at go1.17 (gentraceback) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)

Design. gentraceback is used by panics, runtime.Callers, runtime.Stack and profilers. Inlining forced three changes: inline tables in tracebacks (Go 1.9 781fd3998e), skip counting logical rather than physical frames with skipPleaseUseCallersFrames (ee97216a17), CallersFrames expanding inlined calls (7bf0adc6ad), and then a decision in 1.12 to return PC+1 for inline marks (232c979309) while FuncForPC returns the innermost frame (956879dd0b). LR machines and cgo gave their own edge cases (7de3d71797, e391faded7, ddb503be96).

Timeline.

GDB runtime support

Span: Go 1.0 → Go 1.15 · Status at go1.17: runtime-gdb.py maintained (Python 2/3, gdb 7.12–9.2) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)

Design. src/pkg/runtime/runtime-gdb.py adds pretty-printers for strings, slices, maps, channels and a goroutine/info goroutines command that reads runtime.allgs and each G’s saved sp/pc, loaded automatically from the binary.

Timeline.

Changes in this slice that are not part of a thread are listed on the release pages: 1.0 · 1.1 · 1.2 · 1.3 · 1.4 · 1.5 · 1.6 · 1.7 · 1.8 · 1.9 · 1.10 · 1.11 · 1.12 · 1.13 · 1.14 · 1.15 · 1.16 · 1.17

Runtime: tracing, profiling, metrics and diagnostics (Go 1.18–1.28)

Reader: Claude Sonnet. Cites 447 of 734 commits; 104 uncited commits were judged test-only or cross-area. A few trace-parser items may sit one release off.

The new execution tracer (trace v2): generations, per-M buffers, new parser

Span: Go 1.18 → Go 1.28 · Status at HEAD: default since 1.22 (experiment exectracer2 removed, v1 deleted in 1.23); trace format versions 1.22 → 1.23 → 1.25 → 1.26; flight recorder shipped in 1.25; ongoing hardening and perf work

Earlier history: Execution tracer (trace v1): events, per-goroutine sequencing, annotations (Go 1.5 → Go 1.11)

Design docs & discussion

Design. The old tracer (v1) had one global trace buffer list under trace.lock, per-event timestamps from cputicks, and a parser (internal/trace) that had to load the whole trace in memory and order events by timestamps. The redesign (issue #60773, 43ffe2a892) replaces this with generations. Each M owns two buffer slots (previous/current generation); traceAdvance bumps trace.gen, uses a per-M “write flag” (originally a seqlock, d68aec8db1 made it an atomic.Bool store because only the owner thread writes) to prove every M has observed the new generation, then flushes left-over buffers and emits an EndOfGeneration marker. Every generation is self-contained: all live goroutines get a status event (GoStatus, later GoStatusStack), all strings/stacks are in per-generation tables, and CPU samples are routed to per-generation buffers. Events are one byte of type plus LEB128 varints, decoded by a spec table (tracev2.specs). Ordering does not depend on timestamps: each G and P carries a sequence number written into events to encode a partial order (reset every generation); Ms need none because their buffers are already in order. The runtime-side API is traceAcquire/traceRelease + traceEventWriter (traceruntime.go); runtime/trace.go now carries a long invariants list (251814e580).

The parser side (internal/trace, formerly internal/trace/v2) is a streaming Reader that consumes one generation at a time, orders events with the per-G/P/M sequence numbers (order.go), and exposes typed Events with StateTransition info for Go/Proc state changes; stacks are stored as PCs with sharing (d3d390984b, ~400 MiB saved on a busy Staticcheck trace). It also reads the older formats: Go 1.11–1.19 traces are converted on the fly using Gotraceui’s optimized parser (6133c1e4e2).

Supporting data structures: traceMap was rewritten as an append-only concurrent hash-trie (5bba5b256c) because the fixed bucket table “falls over after a few thousand elements”; its traceRegionAlloc got a lock-free fast path. The tracer was made reentrant by making event writing atomic with respect to stack growth, via a small nosplit core (e76353d5a9), which allowed tracing from anywhere in the runtime. Stack capture uses frame-pointer unwinding (separate thread below).

Timeline.

Frame-pointer unwinding in tracing and profiling; profile stack depth

Span: Go 1.21 → Go 1.27 · Status at HEAD: FP unwinding used by tracer, block and mutex profilers; stack depth 128 default (GODEBUG=profstackdepth); tracefpunwindoff documents which profilers it governs

Design. The tracer walks rbp chains directly (fpTracebackPCs) instead of running the generic unwinder and tracebackPCs, which makes stack capture roughly an order of magnitude cheaper and is the main reason tracing overhead became low. Inline expansion and frame skipping are deferred to report time (fpunwindExpand, refactored to use a provided buffer in b5bfb5a3ce). Where C frames without frame pointers could be on the stack (cgo callback threads), the tracer falls back to the regular unwinder so a registered cgo symbolizer can be used (3dd221a94d). Later, other profilers adopted the same machinery; for the mutex profile deferring skip/inline expansion caused duplicate records with identical stacks, so skipping is now done at record time with full inline expansion for the skipped prefix (87abb4afb6).

Timeline.

Mutex and block profiles, including runtime-internal lock contention

Span: Go 1.19 → Go 1.27 · Status at HEAD: runtime-internal lock contention reported in the mutex profile with sync.Mutex-like semantics (blame the unlocker); runtimecontentionstacks setting removed

Design docs & discussion

Design. sync.Mutex contention is attributed to the unlocker (the critical section that delayed waiters). Runtime-internal lock2 contention was initially attributed to the waiter. The fix (4d0f955a5e, lock_spinbit.go) decides at the start of unlock2 whether to sample; if so it takes a slower unlock path, releases the lock first, then walks the list of waiting Ms (O(N)) to compute the total delay the critical section caused, and reports that with the unlocker’s call stack. Delay is measured in cputicks; zero/negative-delay events are kept (clamped to 0) instead of dropped (1416cebdd5, 433c1d3b4a: choose to drop the zero-delay event when the M’s single saved slot is contested; explicit “have event” flag). _LostContendedRuntimeLock is the placeholder frame when no stack is stored.

Timeline.

Flight recorder and trace subscribers

Span: Go 1.25 → Go 1.26 · Status at HEAD: shipped in runtime/trace (1.25); in-band generation-end signal added 1.26

Design docs & discussion

Design. Because every generation is self-contained (see tracer thread), a circular buffer of recent generations is enough to produce a valid trace on demand. 83df0afc4e ports golang.org/x/exp/trace’s flight recorder into runtime/trace: a subscriber model lets both the ordinary tracer and the recorder consume trace data from the runtime without large runtime changes; byte readers were dropped from batch parsing; the trace clock frequency is read directly from the runtime (considered stable for process lifetime) instead of parsing it from the stream. An in-band end-of-generation signal on the internal ReadTrace (linknamed to runtime/trace) lets subscribers start/stop only at generation boundaries. Follow-ups: SetMinAge test fix (db3e02994c), orders test (c8c3d661b0), 96a6e147b2 marking linknames used by runtime/trace.

Timeline.

CPU profiling, goroutine profiling, pprof labels and the protobuf writer

Span: Go 1.18 → Go 1.28 · Status at HEAD: maintenance; labels in tracebacks behind GODEBUG=tracebacklabels=1

Paths not taken: Background CPU profiling API (never landed) · Hardware-counter (PMU / perf_events) profiling (dormant)

Earlier history: CPU profiler and runtime/pprof writers (legacy format to compressed protobuf) (Go 1.0 → Go 1.15) · Profile labels (Go 1.9 → Go 1.15)

Design docs & discussion

Design. CPU samples flow SIGPROF handler → profBuf (lock-free ring with tags carrying pprof label pointers) → reader goroutine → profileBuilder (runtime/pprof/proto.go) which builds the protobuf, merging inlined fake PCs back into single locations. Label sets were map[string]string; ca63101df4 replaced them with a slice-based LabelSet-backed labelMap and an ordered-keys fast path (geomean −49% time, −52% bytes, set-one 153ns→75ns), partly to let eBPF profilers read labels without map internals. The goroutine profile snapshots every goroutine’s stack with a per-goroutine state machine (absent → in-progress → satisfied) while the world runs; 281cfcfc1b changed system goroutines (finalizer, multiple cleanup goroutines) to participate in the state machine and decide at record time whether to emit a stack.

Timeline.

Goroutine leak profile (GC-based leak detection)

Span: Go 1.26 → Go 1.27 · Status at HEAD: goroutineleak profile on by default; GOEXPERIMENT=goroutineleakprofile removed

Design docs & discussion

Design. Proposal #74609: using the garbage collector to find goroutines blocked forever on channels/sync primitives that no reachable goroutine can unblock. The landing change (8c68a1c1ab) touched ~94 files (runtime channel code, GC marking in mbitmap.go, net/http/pprof, linker support) and registers a new pprof profile type; with the experiment off the profile is simply not registered. Stacks are collected without the goroutine-profile semaphore (8873e8bea2: recorded inline via saveg rather than doRecordGoroutineProfile, which had side-effects on goroutineProfile fields).

Timeline.

runtime/metrics and MemStats

Span: Go 1.18 → Go 1.27 · Status at HEAD: ~dozens of metrics; Read guaranteed not to stop the world (test)

Earlier history: Heap profile accounting and MemStats (Go 1.0 → Go 1.17) · runtime/metrics (Go 1.16 → Go 1.17)

Design docs & discussion

Design. Metrics are described in runtime/metrics/description.go (docs generated from metrics.All) and computed in runtime/metrics.go from metricsSema-protected aggregates (heapStatsAggregate, sysStatsAggregate, gcStatsAggregate, cpuStatsAggregate) with dependency tracking (statDep). Time histograms (timeHistogram) use log-linear buckets: the off-by-one that doubled sub-bucket distance was fixed with a simpler formula (2e9dcb5086), then buckets were coarsened with a minimum bucket bit and fewer sub-buckets for ~3x smaller histograms (87eda2a782). Counters in P-local stats use statsSeq (atomic type, 2883af63c2), consistentHeapStats acquire/release made nosplit (df1837799d).

Timeline.

Coverage redesign: runtime/coverage, internal/coverage, covdata

Span: Go 1.20 → Go 1.24 · Status at HEAD: default since 1.20; coverageredesign experiment removed (41298239cf, Jan 2025)

Design docs & discussion

Design. Instead of one global counter file written by testing, cmd/cover instruments each package with a per-package counter array and a meta-data blob (function units, strings, 128-bit hash). At init, instrumented packages call a registration hook (runtime/covermeta.go; fixed package IDs for runtime, internal/cpu etc. whose functions run before init, cf83a490e4). On exit a runtime exit hook (07bdf1dc54: addExitHook(f, runOnNonZeroExit), run in reverse registration order) writes a meta-data file and a counter-data file into GOCOVERDIR (87db4ffada). Encoders/decoders and a streaming “pod” reader live in internal/coverage (f951f697c4, 84f95aa811, efa3f1749f, 7a74829858); go tool covdata supports merge, intersect, subtract, percent, pkglist, debugdump, textfmt. User APIs runtime/coverage.WriteMeta/WriteCounters/ClearCounters allow capture from long-running programs (ecd112c0d1), restricted to atomic mode because of relaxed-memory-model tearing seen on ppc64 (61e5ea4929).

Timeline.

Race detector, ASan, MSan and Valgrind integration

Span: Go 1.18 → Go 1.28 · Status at HEAD: race on linux/riscv64, loong64, s390x; asan/msan linux incl. loong64, riscv64, ppc64le; build-tag Valgrind support

Earlier history: Race detector integration (Go 1.1 → Go 1.17) · MSan and sanitizer integration (Go 1.6 → Go 1.16)

Design docs & discussion

Design. Each sanitizer needs the runtime to tell it about Go memory. The race detector uses prebuilt runtime/race/*.syso TSAN runtimes and ABI-correct racecall trampolines per arch; ASan wrappers asanread/asanwrite (NOSPLIT) are inserted by the compiler, and the runtime poisons/unpoisons allocations; MSan analogously. Valgrind (40b19b56a9) is gated by a build tag and implemented without cgo or Valgrind headers: a tiny assembly function emits the magic client-request instruction sequence. Memory is modeled as a two-level mempool: heap arenas are a pool created with VALGRIND_CREATE_MEMPOOL_EXT (METAPOOL and AUTO_FREE), spans are “superblocks” (MEMPOOL_ALLOC), and objects are MALLOCLIKE_BLOCK; stacks are annotated too.

Timeline.

Crash output, runtime/debug, BuildInfo and debuglog

Span: Go 1.18 → Go 1.26 · Status at HEAD: debug.SetCrashOutput public; build info includes VCS and settings; debuglog compiled in tests

Earlier history: GODEBUG, runtime/debug, debug log and traceback knobs (Go 1.0 → Go 1.17) · Traceback correctness and inlining (diagnostic output) (Go 1.0 → Go 1.17) · GDB runtime support (Go 1.0 → Go 1.15)

Design docs & discussion

Design. debug.SetCrashOutput(f, CrashOptions) (1bb947b2eb) makes the runtime write fatal panic output also to a file descriptor (crashFD initialized to -1, guarded against FD changes during panic 638b9023e3), so a watchdog child process can record unhandled panics for telemetry. debuglog (runtime/debuglog.go) is a per-M lock-free ring log; originally the debuglog build tag made the logger type-erased through a const. 548158c4a5 makes the tag select a type alias to dloggerImpl or dloggerFake, so the implementation is always compiled and tested (runtime.a +0.9%, no cmd/go size change).

Timeline.

Changes in this slice that are not part of a thread are listed on the release pages: 1.18 · 1.19 · 1.20 · 1.21 · 1.22 · 1.23 · 1.24 · 1.25 · 1.26 · 1.27