Diagnostics and knobs
Thread map, Go 1.0 – 1.17
| Thread | 1.0 | 1.1 | 1.2 | 1.3 | 1.4 | 1.5 | 1.6 | 1.7 | 1.8 | 1.9 | 1.10 | 1.11 | 1.12 | 1.13 | 1.14 | 1.15 | 1.16 | 1.17 | Status at go1.17 (or at the end of its era slice) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Diagnostics and knobs (Go 1.0–1.17) | |||||||||||||||||||
| Execution tracer (trace v1): events, per-goroutine sequencing, annotations | ● | ● | ● | ● | ● | ● | ● | ● | still the v1 design at go1.17 (per-P buffers, text of events decoded by internal/trace, go tool trace UI) → continues as The new execution tracer (trace v2): generations, per-M buffers, new parser (1.18+) | ||||||||||
| Race detector integration | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | still the compiler-instrumentation (racewalk) + TSan-runtime design at go1.17; ports on linux/amd64, darwin, freebsd, windows, netbsd, openbsd/amd64, ppc64le, arm64, darwin/arm64 → continues as Race detector, ASan, MSan and Valgrind integration (1.18+) | ||
| CPU profiler and runtime/pprof writers (legacy format to compressed protobuf) | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | still the design at go1.17: signal-driven SIGPROF samples logged to a profile buffer, aggregated by runtime/pprof goroutines and streamed as gzip protobuf with inline-aware Locations → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+) | |
| Profile labels | ● | ● | ● | ● | ● | still the design at go1.17: g.labels pointer, copied by the signal handler into the CPU profile tag buffer → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+) | |||||||||||||
| Heap profile accounting and MemStats | ● | ● | ● | ● | ● | ● | ● | ● | ● | still the design at go1.17 (consistent stats partly sharded; MemStats copied explicitly), with heapGoal in gcControllerState → continues as runtime/metrics and MemStats (1.18+) | |||||||||
| runtime/metrics | ● | ● | still the design at go1.17 → continues as runtime/metrics and MemStats (1.18+) | ||||||||||||||||
| GODEBUG, runtime/debug, debug log and traceback knobs | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | still the design at go1.17 → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+) | |||
| MSan and sanitizer integration | ● | ● | ● | ● | ● | msan (linux/amd64, arm64) in place; ASan not yet present → continues as Race detector, ASan, MSan and Valgrind integration (1.18+) | |||||||||||||
| GOEXPERIMENT handling (objabi, buildcfg, goexperiment) | ● | regabi sub-experiments on by default in the baseline for amd64 (and Android) → continues in internal/buildcfg/internal/goexperiment (1.18+, see the later report's experiment threads) | |||||||||||||||||
| Traceback correctness and inlining (diagnostic output) | ● | ● | ● | ● | ● | ● | ● | still the design at go1.17 (gentraceback) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+) | |||||||||||
| GDB runtime support | ● | ● | ● | ● | ● | ● | ● | runtime-gdb.py maintained (Python 2/3, gdb 7.12–9.2) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+) | |||||||||||
Thread map, Go 1.18 – 1.28
| Thread | 1.18 | 1.19 | 1.20 | 1.21 | 1.22 | 1.23 | 1.24 | 1.25 | 1.26 | 1.27 | 1.28 | Status at HEAD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Runtime: tracing, profiling, metrics and diagnostics (Go 1.18–1.28) | ||||||||||||
| The new execution tracer (trace v2): generations, per-M buffers, new parser | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | default since 1.22 (experiment exectracer2 removed, v1 deleted in 1.23); trace format versions 1.22 → 1.23 → 1.25 → 1.26; flight recorder shipped in 1.25; ongoing hardening and perf work |
| Frame-pointer unwinding in tracing and profiling; profile stack depth | ● | ● | ● | ● | ● | FP unwinding used by tracer, block and mutex profilers; stack depth 128 default (GODEBUG=profstackdepth); tracefpunwindoff documents which profilers it governs | ||||||
| Mutex and block profiles, including runtime-internal lock contention | ● | ● | ● | ● | ● | ● | ● | runtime-internal lock contention reported in the mutex profile with sync.Mutex-like semantics (blame the unlocker); runtimecontentionstacks setting removed | ||||
| Flight recorder and trace subscribers | ● | ● | shipped in runtime/trace (1.25); in-band generation-end signal added 1.26 | |||||||||
| CPU profiling, goroutine profiling, pprof labels and the protobuf writer | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | maintenance; labels in tracebacks behind GODEBUG=tracebacklabels=1 |
| Goroutine leak profile (GC-based leak detection) | ● | ● | goroutineleak profile on by default; GOEXPERIMENT=goroutineleakprofile removed | |||||||||
| runtime/metrics and MemStats | ● | ● | ● | ● | ● | ● | ● | ● | ~dozens of metrics; Read guaranteed not to stop the world (test) | |||
| Coverage redesign: runtime/coverage, internal/coverage, covdata | ● | ● | ● | ● | ● | ● | default since 1.20; coverageredesign experiment removed (41298239cf, Jan 2025) | |||||
| Race detector, ASan, MSan and Valgrind integration | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | race on linux/riscv64, loong64, s390x; asan/msan linux incl. loong64, riscv64, ppc64le; build-tag Valgrind support |
| Crash output, runtime/debug, BuildInfo and debuglog | ● | ● | ● | ● | ● | ● | ● | debug.SetCrashOutput public; build info includes VCS and settings; debuglog compiled in tests | ||||
Diagnostics and knobs (Go 1.0–1.17)
Reader: Claude Sonnet. Cites 528 of 915 commits in this slice; every non-noise commit is cited, plus 5 larger commits the noise filter had set aside, which a separate gap reader described. Read the design-heavy messages in full and the first lines of the rest. The mutex-profile introduction falls in the scheduler bucket and is covered there.
Release digest for this slice
- Go 1.0 — Everything up to Go 1: runtime/pprof CPU and heap profiling,
runtime/debug,pprof.Profile, GOMAXPROCS/NumCPU, gdb support, in the profiler and pprof threads. - Go 1.1 — Race detector added (compiler racewalk, TSan runtime, Windows port, lazy shadow);
runtime/debugGC controls; heap profile overhead cut from 64kB to 2kB per MB. - Go 1.2 — Block profiler gains select/close support; race instrumentation fixes; “created by” in tracebacks.
- Go 1.3 — Race calls via assembly thunks rather than cgo (about 40-50% faster); profiling and Goexit fixes.
- Go 1.4 — Runtime converted from C to Go reaches profiling (
cpuprof,MemProfile, traceback) and race code. - Go 1.5 — Execution tracer,
go tool traceandruntime/traceintroduced; sequence-numbered ordering; GODEBUGgccheckmarkandmemprofilerate. - Go 1.6 — msan runtime support; tracer made free of write barriers;
debug.SetTraceback; panic frames restored in tracebacks. - Go 1.7 — Per-goroutine sequence numbers cut tracing overhead (453% → 26% at 48 cores); file:line in traces; per-P TSan contexts.
- Go 1.8 — pprof protobuf output for CPU, heap, goroutine and counted profiles; GC events in traces; MemStats documented and
NextGCmade the real heap goal. - Go 1.9 — Streaming protobuf writer, profile labels, inlining-aware tracebacks/
Callers, concurrentruntime.GC(), heap-profile snapshot at mark termination. - Go 1.10 — Block/mutex profiles in new format with delays in ns and mutex scaling; STW trace events; label-race fix.
- Go 1.11 — Trace user annotations (Task/Span/Log),
allocsprofile, race on ppc64le, pprof mapping flags. - Go 1.12 — Race on arm64,
debug.ReadBuildInfo,GODEBUG=madvdontneed, mid-stack inlining traceback work,gctrace=2removed. - Go 1.13 — Runtime
dlogring-buffer debug log;go env -w/-u; arm/mips CPU profile corruption fixed. - Go 1.14 — Inlined function encoding in profiles,
asyncpreemptoffGODEBUG, race support for new timers. - Go 1.15 — Max RSS in heap profiles, goroutine profile labels, block/mutex
secondsparameter, race.sysorebuilds. - Go 1.16 —
runtime/metricsintroduced with histograms; consistent memstats; default toMADV_DONTNEED; race on darwin/arm64; msan memmove. - Go 1.17 — More metrics (scheduling latency, allocs/frees); block profile bias fix; mem profiling disabled when unreachable; race on openbsd/amd64 and register ABI for race calls; GOEXPERIMENT refactor with regabi sub-experiments on by default and experiments in
runtime.Version().
Execution tracer (trace v1): events, per-goroutine sequencing, annotations
Span: Go 1.5 → Go 1.11 · Status at go1.17: still the v1 design at go1.17 (per-P buffers, text of events decoded by internal/trace, go tool trace UI) → continues as The new execution tracer (trace v2): generations, per-M buffers, new parser (1.18+)
- design doc Go Execution Tracer linked from #16619
Design. The tracer was added in one stack of CLs by Dmitry Vyukov from a design document: a runtime part (traceEvent, per-P traceBuf buffers that are flushed as “batches”, a stack table hashed by PC sequence, a string table), runtime/pprof entry points StartTrace/StopTrace, a trace parser in internal/trace, and the go tool trace command. Events are compact varint-encoded records with delta timestamps; a batch event announces the P and base timestamp, and the parser merges batches and restores a consistent order. Stacks and (later) strings are interned into tables that are written at the end of the trace.
Event ordering was the hard problem. In Go 1.5 the first approach was timestamps (cputicks); because of unsynchronised clocks, VMs, non-x86 timer resolution and identical timestamps this produced flaky tests, so the ordering was made explicit with a global event sequence number (80c98fa901) while timestamps were kept for time display. A global sequencer slowed parallel programs by up to 10x, so in Go 1.7 it was replaced by per-goroutine sequence numbers assigned only to three event types (goroutine start, unblock, syscall exit); the parser rebuilds a partial order (a3703618ea: 5026 ns/op without tracing, 27803 with the global sequencer, 6369 with the new scheme at GOMAXPROCS=48; average event size 4.63 → 3.62 bytes). Go 1.7 also embedded file:line info for stack frames into the trace so it no longer needs the binary (0fb7b4cccd), with a version in the trace header that lets the parser still read 1.5 traces (3fafe2e888).
From Go 1.8 the tracer grew GC visibility (mark worker types, mark assists, STW events, sweep statistics) and in Go 1.11 a user annotation API (Task, Region/Span, Log) that bumped the trace version to 1.11 and required locking the string table.
Timeline.
- Go 1.5 — Done: user-visible
runtime: add execution tracing functionality(f30a2b9ca7) withruntime/pprofhooks (986a1d2d1c);cmd/trace(edadffa2f3) moves the parser intointernal/traceand shows trace view, blocking profile, network IO profile and per-goroutine traces; thread id dumped on proc start (9d332a8324) and goroutine analysis moved tointernal/trace(7b0c73aa28); futile wakeups (unblock->start->block) are marked by an event on channels and removed by the parser (4396ea96c4); fixes for GC start/end on different Ps (eec6fdc90b), syscall exit (089d363a91), trace tick frequency on Windows and non-x86 (256116ad25,5f69e739d3), a false race report (4a45ac577f, the race detector is taught thetrace.shutdownSemasynchronisation); explicit sequence numbers (80c98fa901); stable sort and fuzzer fixes (64e48bbaba,e1ee31400a); the API moves fromruntime/pprofto the new user-visibleruntime/tracepackage (ae1ea2aa94). Left open: global sequencer cost on many CPUs (the Go 1.5 commit message says only that time cannot be made reliable); “sync.Mutex will be supported in a separate change” (4396ea96c4). - Go 1.6 — Done: write barriers eliminated from the tracer so it can be called from sysmon and the scheduler:
traceBufPtrreplaces*traceBuf(03227bb55e), the same fortraceStackandtraceAllocBlock(2be1ed80c5,e9aef43d87),traceBufslice replaced by an index (f5c42cf88e); blocking-syscall tracing made identical forentersyscallandentersyscallblock(001a75a74c); hashing of trace stacks fixed (739f133837); stack lock acquired intraceEvent(08594ac7c7). Left open: nothing noted beyond the sequencer cost. - Go 1.7 — Done: per-goroutine sequence numbers (
a3703618ea); file:line in traces (0fb7b4cccd) and parsing of 1.5 traces (3fafe2e888); fixes for timestamp overflow and coarse timestamps (944a0859b9,c4c182140a); trace-event descriptions (2d342fba78); error when the binary is missing for an old trace (c3818e56d0). Left open: StartTrace cost with many goroutines (fixed in 1.8). - Go 1.8 — Done: user-visible GC shown in
cmd/traceon its own row, mark termination labelled (eed309f5fb,94589054d3), mark worker types annotated (6834839427), goroutines blocked on GC assists tracked (6da83c6fc0);StartTraceemits one stack id for all existing goroutines’EvGoCreateinstead of unwinding each (747a158ef3: 10K blocked goroutines 20ms → 4ms); runnable-count andEvGoWaiting/EvGoInSyscallanalysis fixes (c24cc40075,a50f9859bd); global buffer reset inStopTrace(cd285f1c6f);*greplaced byguintptrin the tracer (c242517866); a script to collect canned traces (e14b021977). Left open: nothing noted. - Go 1.9 — Done: mark assists recorded (
2a74b9e814);runtime/traceiterates over frames rather than PCs after inlining (3249cb0ab4); swept/reclaimed bytes recorded in sweep events (22000f5407). Left open: nothing noted. - Go 1.10 — Done: STW GC events traced (
b0392159f6); fix for a corrupted trace inStartTrace: the batch header write moved totraceFlush, andtraceStringnow takes*traceBufPtrso allocation tracing cannot overwrite the buffer (d58f4e9b7b); goroutine info initialised fromGoCreateincmd/trace(a58286c289). Left open: nothing noted. - Go 1.11 — Done: user-visible annotation API proposal that bumps the trace version to 1.11 (
32d1cd33c7) and its implementation: Task/Span/Log eventsEvUserTaskCreate,EvUserTaskEnd,EvUserSpan,EvUserLog, string table guarded bytrace.stringsLock(6977a3b257);internal/tracelinks span/task start and end, with a revert and reland of removing backlinks (1ae22d8cfe,16398894dc,d3562c9db9,d3946f75d3), computes span stats (68a1c9c400,ad638fbfb5,61f92ee56a) and sorts events deterministically (1764609b8b); benchmarks, and Region creation avoided when tracing is off, so pre-existing regions are no longer traced (3bf1644428); old functions deleted (c1492b6bd0,c7519f0dae); package summary reworded (444e7ddb27). Left open: “we can add the feature [tracing pre-existing regions] in the future” (3bf1644428). - Go 1.12 — Done: only maintenance:
internal/traceparserreverted incmd/trace(6d5caf38e3, Go 1.12). Left open: no new tracer features; the v1 design stays through 1.17.
Race detector integration
Span: Go 1.1 → Go 1.17 · Status at go1.17: still the compiler-instrumentation (racewalk) + TSan-runtime design at go1.17; ports on linux/amd64, darwin, freebsd, windows, netbsd, openbsd/amd64, ppc64le, arm64, darwin/arm64 → continues as Race detector, ASan, MSan and Valgrind integration (1.18+)
Design. The race detector (Dmitriy Vyukov) is the ThreadSanitizer (TSan) runtime linked as .syso files in runtime/race, driven by compiler instrumentation. Under -race cmd/gc/racewalk.c rewrites the AST so that every memory access calls runtime.raceread/racewrite (or the range variants for aggregates), function entry/exit call racefuncenter/racefuncexit, and the runtime calls acquire/release annotations from channels, locks, sync/atomic, goroutine creation and finalizers. Instrumentation is skipped for non-escaping, blank, and zero-sized values and for runtime-internal helpers.
Throughout 1.1 the compiler pass was hardened case by case (nested structs, selectors, arrays, &&/||, for conditions, type switches, append, OKEY, OEFACE), and its cost was reduced: racefuncenter takes its PC directly (ce287933d6, about 46-48% faster on encoding/json), and in 1.3 the cgo-based calls were replaced by custom assembly thunks that extract the caller PC, check that the address is heap or global and switch to the g0 stack (a1695d2ea3: regexp 3.69s → 2.23s, bzip2 9.46s → 4.70s, json 6.38s → 3.63s). In 1.4–1.5 the runtime was converted to Go, so the race detector had to ignore accesses on g0/gsignal stacks (467a6d28be) and race.c became Go (59e3e5354d). In 1.7 the TSan runtime was given per-P contexts, which fixed unbounded growth of sync object descriptors (caa2147532; an alternate procwire scheme was rejected as too messy).
Timeline.
- Go 1.1 — Done: user-visible race detector for linux/darwin/freebsd amd64 and Windows, runtime hooks (
2f6cbc74f1); lazily allocated shadow memory, necessary on Windows and removing a 16TB virtual size (1a19f01a68,1ebf2d85ba); data/bss shadow mapping alignment (12b7db3d57);racewalkrewrite: block instrumentation (c46f1f40da), init lists (b11f85a8aa), refactor to a generic walk (703043c8dc), no double evaluation of function calls (abb313f8c8), params returned from heap (a3a7244779), structs and itabs (96833d3a25,89bfddbf67,db8d7a292d), more node kinds and&&/||/HMUL/append/type switch (656bc3eb96,c0b3c17184,eb7c51c148,5f9a9433ea,7c79910cb9,ecbf99ad97,ca4b868e9a); direct PC toracefuncenter(ce287933d6) andruntime·callerswhen the PC is on the heap (f59a605645);racefuncenterregister/arg fixes (ec892be1af,e2f9e816b7); nosplit callbacks (0ead18c59e); handling of stackalloc (27e93fbd00); finalizer-aware detection (063c13a34c);RaceRead/RaceWriteAPI in the runtime (51e89f59b2) and range access functions (ccc61eadd5); better stack traces: runtime frame on top (0ce96f9ef4) and sync/atomic frames (c3c107f67c). Left open: the race runtime depends on cgo (removed in 1.3 via thunks). - Go 1.2 — Done:
Read/Writesyscalls annotated with the memory they touch (fc80764792); instrumentation of for loops, arrays, selectors, type switches fixed (591d58a3bb,3be794cdc2,6c4943cb51,a15074c4dc,e7657de717);forkAndExecInChildexcluded since instrumentation may allocate and preempt (63e0ddc7bf); no preemption of race calls (d017f578d0); unusedstepparam removed (e2d95c1f24). Left open: nothing noted. - Go 1.3 — Done: custom assembly thunks for race calls instead of cgo (
a1695d2ea3), which also removes special cases fromcgocalland the scheduler; kind comparison ignoresKindNoPointers(f59ea4e58b);forpost-condition in -race (e56dc99665). Left open: “In long-term it will allow to remove cyclic runtime/race dependency on cmd/cgo” (a1695d2ea3). - Go 1.4 — Done: runtime converted to Go-compatible race handling, ignoring accesses on g0/gsignal (
467a6d28be); writebarrier dst argument instrumented (d78823168f); undefined behaviour inracewalk.cfixed (1f5800673b). Left open: nothing noted. - Go 1.5 — Done: race implementation converted from C to Go [dev.cc] (
59e3e5354d);appendinstrumentation, stack corruption and PC adjustment fixes (ed8cc5cf9b,f59f9b8527,3fc529eabe); write barriers no longer instrumented (972a478ddf);OGETGandOKEYhandled (08c43488ee,cd0a8ed48a). Left open: nothing noted. - Go 1.6 — Done:
race1.gomerged intorace.go(243757576d); instrumented calls have a type width (9179c9cb5c). Left open: nothing noted. - Go 1.7 — Done: per-P TSan contexts (
caa2147532);racefinionce only (733162fd6c); atomic ops on external memory (bb337372fb); OASWB rewriting (595426c0d9); a TSan memory leak (5.6GB after one day, 12.2GB after six) fixed by rebuilding the.sysofiles (d73ca5f4d8). Left open: nothing noted. - Go 1.8 — Done: copy/append not instrumented inside the runtime (
9d8522fdc7); no crash on invalid PCs (38765eba73); code duplication in instrumentation (8b3194ac8f). Left open: nothing noted. - Go 1.10 — Done: false positive when reading profile labels fixed with
racereleasemergeon alabelSyncglobal (57584a0ee1, 1.10; alsoc82efb1fa31.9, 1.9); no race calls for zero-sized values andraceXinstead ofraceXrangefor types without subcomponents (b046878018,53c8be4a8d);Posleft unset for racewalk enter/exit (d2414cefba). Left open: nothing noted. - Go 1.11 — Done: user-visible ppc64le support with
race_ppc64le.sandracefuncenterfp(9e9ff565cd, 1.11). - Go 1.12 — Done: user-visible user-visible arm64 (
978cfa8e46, 1.12) with arg maps for sync/atomic functions (c6e698d5dd). Left open: nothing noted. - Go 1.13 — Done: arm64 uses
raceprocctx(6e37e3a80b). - Go 1.14 — Done: race support for new timers (
ab3f1a23b6); several runtime frames per TSan PC for mid-stack inlining (a14efb1be3, still needs a rebuilt tsan library); arm64 g register saved across callbacks (ab81efa3dc); ppc64le fix (44e752c38a). - Go 1.15 — Done:
.sysorebuilds for netbsd and withoutgetauxval(3afa74115b,fffe622963,e49b2308a5). Left open:a14efb1be31.14 says the tsan library and.sysofiles must still be rebuilt to complete #33309. - Go 1.16 — Done: user-visible darwin/arm64 (
b53df56001);raceacquire/racereleaseorder swapped in chan send/recv and select to follow the memory model (35455fff0e); arg maps for sync/atomic (0932dc2118); instrumentation no longer clobbers the frame pointer (8c39bbf9c9). Left open: nothing noted. - Go 1.17 — Done: user-visible openbsd/amd64 (
b3b65f2176); register ABI:racecallbackthunkABIInternal, X15 zeroed inracecall,BXinstead of R15, andraceread/racewrite/racewriterangeuse the actual internal ABI (e79c2fd428,946351d5a2,a4dac8bd22,0a510478b0);racefuncenterfpremoved (87d29939c8). Left open: nothing noted.
CPU profiler and runtime/pprof writers (legacy format to compressed protobuf)
Span: Go 1.0 → Go 1.15 · Status at go1.17: still the design at go1.17: signal-driven SIGPROF samples logged to a profile buffer, aggregated by runtime/pprof goroutines and streamed as gzip protobuf with inline-aware Locations → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+)
Paths not taken: Background CPU profiling API (never landed)
- proposal #16093 runtime/pprof: generate profiles in gzipped proto format referenced by 5 of this thread's commits
Design. CPU profiling began (2011) as a hash table in the signal handler (cpuprof) with an eviction log and a legacy binary format read by the pprof tool. Go 1.4 converted cpuprof from C to Go (ac49e6735b) and 1.5 simplified it (3c8a89daf3, 481ce36ec4). The profile format was replaced in Go 1.8: first conversion code from legacy to protobuf (7d14401bcb, f72a629dbd), then CPU (76f12cdaa2), count profiles (ccd69d0582), heap (86ab09eed5) and goroutine locations (c6228ef7e2) were emitted as gzip-compressed pprof protobuf at debug=0. The first implementation buffered the profile three times (in-memory profile, struct representation, marshalled bytes); Go 1.9 added a streaming protobuf encoder so only one copy exists (cbab65fdfa) and switched CPU profiling to a plain sample log (“new profile buffers”, 1a680a902a), with hash-based dedup done in an ordinary goroutine, a more efficient staging hash table (1564817d8c) and symbolization (cbef450df7). Inlining-aware encoding followed (Go 1.9: 9e83c11fca; Go 1.14 corrected inlined functions into one Location with multiple Lines, e038c7e418, e25de44ef2, 71c59ee6fc, and later fadbf7404d expanded the final truncated frame so locations are not split).
Timeline.
- Go 1.0 — Done:
- 2009: user-visible
runtime.GOMAXPROCS(7955490de2). - 2010: malloc sampling and a pprof interface (
6eb251f244), heap profile extra information (baf538406b),MemStatsType(87915b6565). - 2011: user-visible
runtime/pprofCPU profiling (b47ec598b7), pprof deadlock fix (bed7e3ed78), OS X documented as broken (d0eaa58b5a);runtime/debugpackage for stack traces (89993544c2). - 2012: user-visible
pprof.Profiletype for custom profiles (ebae73bb24);runtime.NumCPU(6dfdd4c1e3,0ae9d812c8,610757b155removes a duplicate pcln walker). Left open: nothing noted.
- 2009: user-visible
- Go 1.1 — Done: heap profile lowers memory overhead from 64kB to 2kB per MB of address space by using a 4096-byte interval (
baf91c313f); total-allocation profile corrected by including zero-in-use buckets (c4a814f2da); end-of-log marker in CPU profile (b2458ff75c); line numbers adjusted to call sites (318309a51f); memory profiles allowed with GOGC=off (0c3b17a55a). Left open: nothing noted. - Go 1.2 — Done: user-visible block profiler supports select, channel close and shows the top frame in debug mode (
c92287686d);SetBlockProfileRatecorner cases fixed (3bd0b0a80d); memory profiler usespersistentalloc(8cf7044983); “created by” printed for running goroutines (c33d490020). Left open: nothing noted. - Go 1.3 — Done: separate large and small objects at the same stack in the mem profile (
e71d147750); stopping CPU profile when off (aeeda707ff);GoroutineProfilecrash (bc135f6492); divide in profiling tracebacks on ARM (2c98a3bc2e); specials/proflock deadlock fix (b039abfc3e). Left open: nothing noted. - Go 1.4 — Done:
MemProfile,BlockProfile,ThreadCreateProfileconverted to Go (04487d5612); cpuprof converted from C to Go (ac49e6735b); data race in pprof double-checked locking fixed (fe7b29f5fd); profileBucketsub-structs named (a9cef4952f); no tracing pastjmpdeferon arm (597f87c997). Left open: nothing noted. - Go 1.5 — Done: CPU profiler simplified (
3c8a89daf3,481ce36ec4); heap statistics always written (c26fc88d56,79986e24e0) and heap profile content documented (6b8762104a);GODEBUG=memprofilerate(see knobs,3c4be235be,027e47b7b1);runtime.callerswalks the calling G (719efc70eb). Left open: nothing noted. - Go 1.6 — Done: SIGPROF handled inside stack barriers (
3a2fc06833);GoroutineProfilefixes (313fd1cb13,d9fdbf4820); documentation ofSetCPUProfilewith c-archive/c-shared (7688ffe134). Left open: nothing noted. - Go 1.7 — Done: counted profiles sorted by count (
6969d9bf03);runtime.CallersFrames/Framesintroduced for file and line (ad03af66eb); PIE profiling (4223294eab);ThreadCreateProfilestack copying simplified (756ea30eb0). Left open: CPU profiles are stored in memory in full. - Go 1.8 — Done: user-visible profile protobuf format:
protopprofconversion (f72a629dbd,7d14401bcb,b33030a727,21a1fe4ddf,eafe48781aremoves the gzip0 package), CPU (76f12cdaa2), counts (ccd69d0582), heap (86ab09eed5), goroutine locations (c6228ef7e2); heap profile reports GCSys/OtherSys (39ce6eb9ec). Left open:f72a629dbd/76f12cdaa2read and convert the whole profile in memory, “should be possible to convert each chunk in a stream”. - Go 1.9 — Done: streaming encoder with one copy in memory (
cbab65fdfa); new profile buffer for CPU (1a680a902a); hash table for staging (1564817d8c); symbolized proto profiles (cbef450df7) with inlined frames expanded (9e83c11fca); GNU build IDs in mappings (c797256a8f) and a dummy huge page mapping ignored (3ddf65015a); period information fixed (0c0c94a9dc);Profile.Addrace fixed (9b15c13dc5); empty stacks handled (c7894924c7); mips SIGPROF deadlock prevented (835dfef939); newly addedMemStatsfields printed (8833af3f4b). Left open: “get back down to storing it zero times (streaming it to the underlying io.Writer)” (cbab65fdfa). - Go 1.10 — Done: block/mutex profiles use the new profile format with delay in nanoseconds (
f99d14e0de), mutex profile scaled by sampling rate (e89d08e021; block profile not scaled because “how to infer unsampled values is unclear”); memstats read earlier (301b127a05); lost extra reset (96b1eff087); unwinding through cgo hardened (5c2be42a68,ebe38b867c,3ff41cdffa). Left open: block profile scaling. - Go 1.11 — Done: user-visible
allocsprofile (cd037bce09); mappings setHasFunctions(7b2f55d89f) with a fake mapping when/proc/self/mapsis unavailable (ba02264446);SetMutexProfileFractionnegative values specified (3d8940a9ee). Left open: nothing noted. - Go 1.12 — Done: memory profile block size computed from sampled values (
f9a4ae018d); “deleted” suffix stripped from maps (73b8e5f81b). - Go 1.13 — Done: CPU profile corruption on arm/mips fixed — lost atomic64 count was racy (
91c385b393). Left open: nothing noted. - Go 1.14 — Done: inlined functions encoded as one Location with several Lines for CPU and non-CPU profiles (
e038c7e418,e25de44ef2,71c59ee6fc,45b4ed7577); fewer allocations buildingLabelSet(5fb74fc138); crash on truncated stacks avoided (ea18a1c212). Left open: nothing noted. - Go 1.15 — Done: user-visible max RSS in memory profiles (
fb1cd94222, units fixed197a2a3799); goroutine profile plumbs labels (38c2c12bc1);net/http/pprofsecondsparameter for block and mutex, viainternal/profilemerge (2ff1e3ebf5); faster CPU profile shutdown (1f0738c157); final stack frame expansion (fadbf7404d); fake overflow record PC (4a8b9bd264); recursive inline heuristic (ce52871948); mcaches flushed before reading memstats (0ddde4ada2). Left open: nothing noted. - Go 1.17 — Done: user-visible block profile bias corrected by aggregating short events as longer, less frequent events (
0ae9c3b98b, 1.17); memory profiling is disabled whenruntime.MemProfileis unreachable in the linked binary (18510ae88f, 1.17);cpuprofhandles nil gp (c80da0a33a);writeProfileInternalcommon code (79d03ad739). Left open: nothing noted.
Profile labels
Span: Go 1.9 → Go 1.15 · Status at go1.17: still the design at go1.17: g.labels pointer, copied by the signal handler into the CPU profile tag buffer → continues as CPU profiling, goroutine profiling, pprof labels and the protobuf writer (1.18+)
- design doc Proposal: Support for pprof profiler labels Michael Matloob, 15 May 2017 (to reflect actual implementation) · linked from #17280
- proposal #17280 pprof: add support for profiler labels referenced by 1 of this thread's commits
Design. Labels (pprof.WithLabels, Labels, Label, ForLabels, Do, SetGoroutineLabels) attach key/value pairs to goroutines. The runtime stores an opaque pointer labels in g, inherited at goroutine creation, set via runtime_setProfLabel and read by the SIGPROF handler into the CPU profile buffer; the proto writer then propagates them into samples. Because the signal handler cannot use real acquire/release, a global labelSync models happens-before for the race detector.
Timeline.
- Go 1.9 — Done: user-visible label types (
91ad2a2194),SetGoroutineLabelsandDowith runtime support (62956897c1), labels propagated into the profile proto (f105c91757), race-detector annotations (c82efb1fa3). Left open: nothing noted. - Go 1.10 — Done: false-positive race when reading labels fixed with
racereleasemerge(57584a0ee1). Left open: nothing noted. - Go 1.13 — Done: heritability of labels documented (
0ca4f6be35,9a71015860“documents labels bug”). - Go 1.14 — Done: fewer allocations building
LabelSet(5fb74fc138). - Go 1.15 — Done: labels in goroutine profiles via
goroutineProfileWithLabels(38c2c12bc1, 1.15). Left open: OS-thread creation profiles do not capture labels (38c2c12bc1).
Heap profile accounting and MemStats
Span: Go 1.0 → Go 1.17 · Status at go1.17: still the design at go1.17 (consistent stats partly sharded; MemStats copied explicitly), with heapGoal in gcControllerState → continues as runtime/metrics and MemStats (1.18+)
Design. MemStats is a snapshot assembled by updatememstats stopping the world. The heap profile is built from mProf buckets attached to sampled allocations by “specials” on spans; sampling is exponential with MemProfileRate. In Go 1.9 the profile was made cycle-consistent: a global heap profile cycle counter is incremented during mark termination (cheap snapshot) and cleaned later (eee85fc5a1, 3ebe7d7d11, diagrammed in 673a8fdfe6) so a free cannot be attributed to an earlier cycle than its alloc. In Go 1.16 next_gc, stacks_inuse, gcWorkBufInUse etc. were turned into consistent per-P sharded stats (f77a9025f1, ae585ee52c) in preparation for runtime/metrics; the heap goal then moved into gcControllerState (3eaf75c13a).
Timeline.
- Go 1.0 — Done:
MemStatsType(87915b6565) and a malloc sampling profile (6eb251f244). Left open: nothing noted. - Go 1.5 — Done: user-visible GC CPU utilisation in MemStats (
23e4744c07);gctraceformat documented (4c9464525e);GODEBUG=gccheckmark(654297cb02, see knobs). Left open: nothing noted. - Go 1.6 — Done: GC/MemProfile interaction documented (
08b80ca880). Left open: nothing noted. - Go 1.7 — Done: scavenger memory summary documented (
77026ef902). Left open: nothing noted. - Go 1.8 — Done:
MemStatsdocumented (f67c9de656,a3f4cc0669);next_gcdisentangled from the trigger soNextGCis the heap goal andgc_triggeris unexported (ec9c84c884),heap_reachableremoved (2098e5d39a),next_gcis ^0 when GC is off (fa9b57bb1d), 8-byte alignment ofheap_live(d3a3b74aa1); user-visible number of forced GCs added (01c6a19e04). Left open: nothing noted. - Go 1.9 — Done: user-visible
runtime.GC()triggers a concurrent GC that still completes a full cycle and publishes the heap profile (9ffbdabdb0); heap profile cycle type and snapshot at mark termination (3ebe7d7d11,eee85fc5a1); manually-managed spans not counted inheap_inuse/sys(dc0f0ab70f); atomic loads ofheap_live(9d36163c0b). Left open: nothing noted. - Go 1.15 — Done: mcaches flushed to mcentral before reading memstats (
0ddde4ada2). Left open: nothing noted. - Go 1.16 — Done:
next_gcatomically accessed (f5c6875f32); MemStats copied field by field (39e335ac06);heap_allocremoved (ae585ee52c); stacks, work-buf and prog-bits stats replaced by consistent stats (f77a9025f1);sysMemStatsmethods nosplit (22d2b984a6);allocs-by-sizebucket fix (8db7e2fecd). Left open: nothing noted. - Go 1.17 — Done:
next_gc/last_next_gcmoved togcControllerStateasheapGoal/lastHeapGoal(3eaf75c13a). Left open: nothing noted.
runtime/metrics
Span: Go 1.16 → Go 1.17 · Status at go1.17: still the design at go1.17 → continues as runtime/metrics and MemStats (1.18+)
- design doc Proposal: API for unstable runtime metrics Michael Knyszek · linked from #37112
- proposal #37112 runtime: API for unstable metrics referenced by 7 of this thread's commits
Design. runtime/metrics exposes named metrics (/gc/heap/allocs:bytes style, with unit suffix) via All() descriptions and Read([]Sample), where the value is a Value union (uint64, float64, *Float64Histogram). Implementation in runtime/metrics.go computes values from sharded consistent stats and the runtime’s timeHistogram, a concurrent HDR-style histogram with log-linear buckets (36c5edd8d9). Histogram semantics were changed before release: shifted buckets to allow negative durations (b116404444), explicit boundaries rather than implicit infinite buckets (ae97717133), units aligned with bucket unit (32afcc9436), marked cumulative (6250833911).
Timeline.
- Go 1.16 — Done: user-visible package interface (
2159c26ceb); memory metrics and baseRead(b08dfbaa43); heap object count (07c3f65d53); heap goal and GC cycle count (a8b28ebc87); allocation and free size distributions (8e2370bf7f);timeHistogramand GC pause distribution (36c5edd8d9,d39a89fd58); goroutine count (80c6b92ecb); wording tweaks (48a1a51898);Readcaveats documented (e0d20e52ee); bucket fixes (8db7e2fecd,b116404444); explicit boundaries (ae97717133); unit and cumulative flags (32afcc9436,6250833911); panic message (4c668b25c6). Left open: metric set is partial relative to MemStats (filled in 1.17). - Go 1.17 — Done: user-visible scheduling latency distribution (
bedfeed54a, approximation by sampling); additional allocation metrics/gc/heap/allocs:bytes|objectsand frees, fillingTotalAlloc(897baae953); tiny allocs metric (0b9ca4d907). Left open: nothing noted.
GODEBUG, runtime/debug, debug log and traceback knobs
Span: Go 1.0 → Go 1.17 · Status at go1.17: still the design at go1.17 → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)
Design. GODEBUG is parsed by the runtime into a table of dbgvar entries (name, address); GOTRACEBACK was extended and made settable from code. Each transition below is a separate commit.
Timeline.
- Go 1.1 — Done: user-visible
runtime/debugGC controls:SetGCPercent,FreeOSMemory,ReadGCStats(472354f81e);debug.Stackdeprecated in favour ofruntime.Stack(e49a183b76). Left open: nothing noted. - Go 1.2 — Done: thread count limit (
665feeedcb); m stack shown on crash (ab38e2a498). Left open: nothing noted. - Go 1.3 — Done:
SetGCPercentforces a GC to be effective (0db71338ed);Stackoutput fix for dotted package paths (d7c14655a9); up to 10 argument words printed (c40480ddd9). Left open: nothing noted. - Go 1.4 — Done: functions of
runtime/debugconverted to Go (ef50462378); exported runtime functions shown in stack traces (54245cba1f);gentracebackof self on the user stack forbidden because it may grow the stack (39bcbb353c);Stackfixes (a8c5e87b1e,9a5654abef). Left open: nothing noted. - Go 1.5 — Done: user-visible
GODEBUG=gccheckmark=0/1(default on during 1.5 dev) replacing runtime functions (654297cb02);GODEBUG=memprofilerate(3c4be235be, renamed027e47b7b1); stack of G printed during signal (4b78c9575d). Left open: gccheckmark “will probably remain [enabled] for much of the 1.5 development cycle”. - Go 1.6 — Done: user-visible
debug.SetTraceback(bb0567b304); panics shown in tracebacks again,gopanicrewritten topanic(0c02bc009a);GODEBUGdoc pointers to net and net/http (cd91c3b036);debug.Stackviaruntime.Stack(2d697b2401). Left open: nothing noted. - Go 1.8 — Done: runtime panic frame elided at top of user stack (
f9feaffdf5). Left open: nothing noted. - Go 1.9 — Done:
SetGCPercentno longer triggers a GC when negative or otherwise (9f75ecd5e1,227fff2ea4); inlined frames in tracebacks andruntime.Callers(781fd3998e,ee97216a17,7bf0adc6ad); endless-recursion tracebacks print the top and bottom 50 frames (3a813386221.16, listed under 1.16 in the log). Left open: nothing noted. - Go 1.10 — Done: hexdump printed on traceback failure (
dbd8f3d739); autogenerated wrappers hidden (e97209515a,032678e0fb). Left open: nothing noted. - Go 1.12 — Done: user-visible
debug.ReadBuildInforeturns module information (45e9c5538b);GODEBUG=madvdontneed=1(22738f07c8);gctrace=2removed (edc2d17086); proper panic tracebacks with mid-stack inlining andFuncForPCreturning the innermost frame (af134b17da,956879dd0b,232c979309). Left open: nothing noted. - Go 1.13 — Done: binary ring-buffer debug log
dlogper-M, compiled only with thedebuglogbuild tag, printed on panic (3ebb1ad9cd); TLS 1.3GODEBUGdocumented (48bb611667); user-visiblego env -w/-ufor persistent environment (f0e9754696). Left open: “decoding from a core dump hasn’t been implemented” (3ebb1ad9cd). - Go 1.14 — Done: user-visible
GODEBUG=asyncpreemptoff=1added before it does anything (40b7455877). Left open: “doesn’t do anything yet” in that commit. - Go 1.15 — Done: debuglog traceback fix (
45cd312394);invalidptrdocumented (3c35eec618). Left open: nothing noted. - Go 1.16 — Done: user-visible default back to
MADV_DONTNEEDon Linux (05e6d28849), afterMADV_FREEfrom 1.12 gave confusing stats; allocation of too-big stacks error improved (331614c4da); stack trace elision uses inlined function name (e8de596f04). Left open: nothing noted. - Go 1.17 — Done:
GODEBUG=scavengemode removed (9c54f878d2);mcall,morestack,asmcgocallno longer treated as top of stack (5ecd9e34df,776ee4079a,a54f7fc0fd), SPWRITE ignored in syscall functions (02e5a8fdfc). Left open: nothing noted.
MSan and sanitizer integration
Span: Go 1.6 → Go 1.16 · Status at go1.17: msan (linux/amd64, arm64) in place; ASan not yet present → continues as Race detector, ASan, MSan and Valgrind integration (1.18+)
Design. -msan makes the compiler insert msanread/msanwrite and the runtime wrap allocations with msanmalloc/msanfree, calling the C MSan runtime via runtime/msan through cgo-free thunks. Instrumentation is skipped on the system stack and when g or m is nil (early init).
Timeline.
- Go 1.6 — Done: user-visible msan runtime support, with calls inserted by
-msan(5174df9087);msanreadskipped on the system stack (880a689124). Left open:msanmalloc/msanfree“will be from other runtime functions in a subsequent CL”. - Go 1.8 — Done: msan calls before traceback functions (
d00890b5f3); nil g/m check inmsanread(ea7d9e6a52). Left open: nothing noted. - Go 1.11 — Done: NetBSD thread sanitizer support in
runtime/cgo(f5921d48f1). Left open: nothing noted. - Go 1.14 — Done: nil g checked in arm64
msancall(827584e2f3). Left open: nothing noted. - Go 1.16 — Done:
__msan_memmoveused for moving data,msanreadsplit to fields (07cba70d57); dummymsanmove(eb21b31e48). Left open: nothing noted.
GOEXPERIMENT handling (objabi, buildcfg, goexperiment)
Span: Go 1.17 → Go 1.17 · Status at go1.17: regabi sub-experiments on by default in the baseline for amd64 (and Android) → continues in internal/buildcfg/internal/goexperiment (1.18+, see the later report’s experiment threads)
Design. GOEXPERIMENT is a comma-separated list; the parser lives in internal/buildcfg with generated boolean and integer constants in internal/goexperiment (b675e52e95 consolidates them, e.g. goexperiment.RegabiWrappersInt). The baseline set depends on GOOS/GOARCH so it moved out of goexperiment into objabi (cf2396c70e) and then buildcfg. Regabi is an alias that turns all sub-experiments on, and noregabi all off (14dbd6e776), not a flag in itself. The set of non-default experiments is recorded in runtime.Version() and go version (5159c83641).
Timeline.
- Go 1.17 — Done: user-visible constants consolidated (
b675e52e95); baseline moved to objabi (cf2396c70e); regabi sub-experiments enabled by default in turn:regabiwrappers(fd3612e433),regabig(bbb510ccc9),regabireflect(88655480f3),regabidefer(af8a176e91),regabiargs(9dd71ba913);regabian alias (14dbd6e776) and enablingregabiargs(43466399cb); Android baseline included (9c12f1b433); bad GOEXPERIMENT setsErrorinstead of panicking (e15d1f4e64);ParseGOEXPERIMENThelper forgo envvalidation (c95464f0ea); experiments reflected inruntime.Version()(5159c83641). Left open:go envGOEXPERIMENT support was the next step (c95464f0ea).
Traceback correctness and inlining (diagnostic output)
Span: Go 1.0 → Go 1.17 · Status at go1.17: still the design at go1.17 (gentraceback) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)
Design. gentraceback is used by panics, runtime.Callers, runtime.Stack and profilers. Inlining forced three changes: inline tables in tracebacks (Go 1.9 781fd3998e), skip counting logical rather than physical frames with skipPleaseUseCallersFrames (ee97216a17), CallersFrames expanding inlined calls (7bf0adc6ad), and then a decision in 1.12 to return PC+1 for inline marks (232c979309) while FuncForPC returns the innermost frame (956879dd0b). LR machines and cgo gave their own edge cases (7de3d71797, e391faded7, ddb503be96).
Timeline.
- Go 1.1 — Done: LR saved on panic for leaf functions (
7de3d71797, 1.1). - Go 1.2 — Done: traceback/GC boundary (
e58f798c0c), arg printing (eac6bee7c1), Windows (0a3bd045f51.3, 1.3), up-to-10 args (c40480ddd91.3). Left open: nothing noted. - Go 1.4 — Done: traceback converted from C to Go (
fa2af441f1); arm trap and unexpected-pc fixes (d889f5f01e,e5e547c71f);newproc1on M stack (6ed58c2962). Left open: nothing noted. - Go 1.7 — Done: traceback stops at foreign function (
88d3db0a5b), cgo traceback PC and signal context (c08436d1c8,3d037cfaf8,03abde4971). Left open: nothing noted. - Go 1.9 — Done: accurate inlined tracebacks and
Callers(781fd3998e,ee97216a17,7bf0adc6ad,e4f73769bc). Left open: mid-stack inlining not yet enabled. - Go 1.12 — Done:
getArgInfofixes (d5bd797ee51.8 1.8; fast path13cd0061391.11, 1.11), defers on LR machines (e391faded71.11),isSystemGoroutinestable mode (29b21ec4c3),gentracebackwrong g and morestack transition (c9986d1452,ae6361e4bd), inline-aware panic tracebacks (af134b17da) and encouragement of single-call-body inlining (13baf4b2cd). Left open:systemstackframe not included in the traceback (ae6361e4bd). - Go 1.17 — Done: arg maps for unstarted goroutines (
38c129b4f01.13), multiple frames per PC for tsan (a14efb1be31.14), faulting-line fix (9ee6ba089d1.14), top-of-stack handling (5ecd9e34df,776ee4079a,a54f7fc0fd). Left open: nothing noted.
GDB runtime support
Span: Go 1.0 → Go 1.15 · Status at go1.17: runtime-gdb.py maintained (Python 2/3, gdb 7.12–9.2) → continues as Crash output, runtime/debug, BuildInfo and debuglog (1.18+)
Design. src/pkg/runtime/runtime-gdb.py adds pretty-printers for strings, slices, maps, channels and a goroutine/info goroutines command that reads runtime.allgs and each G’s saved sp/pc, loaded automatically from the binary.
Timeline.
- Go 1.0 — Done: gdb debug info for builtin structured types and pretty printing (
9a71bb00bb); goroutine support and fixes for strings, channels and missing types (db22e236fd,43512e6c70,46ed89b7a3,8a4ef5d1eb,fb2706113f). Left open: nothing noted. - Go 1.1 — Done:
parse_and_evalfor goroutine id (ca8aac698f). - Go 1.2 — Done: explicit init of helpers (
927b7ac327, 1.2). - Go 1.3 — Done: partial Python 3 support (
f12a167ba2). Left open: nothing noted. - Go 1.5 — Done: loading,
SliceValue, rtype and map printers, Python 2 (a25af2e99e,545686857b,98651d6edf,02d80b9e93,1c82e236f5,53840ad6f1). Left open: nothing noted. - Go 1.11 — Done: sp set before pc (
cbc26869b71.7), goroutine N for running goroutines and tab fixes (24659717811.9,a7d7d7aec21.9), atomicstatus-based state and constants from the runtime (dc3bef3635,ef175731ff,ef400ed20a), sp switching (048988a0f11.12),goroutine all(8c1f78524e1.13). Left open: nothing noted. - Go 1.15 — Done: repair of slice/string/channel printing for several gdb versions (
ee776b4126,3949cae441,429d2c548d). Left open: nothing noted.
Changes in this slice that are not part of a thread are listed on the release pages: 1.0 · 1.1 · 1.2 · 1.3 · 1.4 · 1.5 · 1.6 · 1.7 · 1.8 · 1.9 · 1.10 · 1.11 · 1.12 · 1.13 · 1.14 · 1.15 · 1.16 · 1.17
Runtime: tracing, profiling, metrics and diagnostics (Go 1.18–1.28)
Reader: Claude Sonnet. Cites 447 of 734 commits; 104 uncited commits were judged test-only or cross-area. A few trace-parser items may sit one release off.
The new execution tracer (trace v2): generations, per-M buffers, new parser
Span: Go 1.18 → Go 1.28 · Status at HEAD: default since 1.22 (experiment exectracer2 removed, v1 deleted in 1.23); trace format versions 1.22 → 1.23 → 1.25 → 1.26; flight recorder shipped in 1.25; ongoing hardening and perf work
Earlier history: Execution tracer (trace v1): events, per-goroutine sequencing, annotations (Go 1.5 → Go 1.11)
- design doc Execution tracer overhaul linked from #60773
- proposal #60773 runtime/trace: execution tracer overhaul referenced by 10 of this thread's commits
- tracking issue #66703 runtime/trace: set the v2 tracer as the default tracer referenced by 2 of this thread's commits
- background Go Execution Tracer linked from #60773
Design. The old tracer (v1) had one global trace buffer list under trace.lock, per-event timestamps from cputicks, and a parser (internal/trace) that had to load the whole trace in memory and order events by timestamps. The redesign (issue #60773, 43ffe2a892) replaces this with generations. Each M owns two buffer slots (previous/current generation); traceAdvance bumps trace.gen, uses a per-M “write flag” (originally a seqlock, d68aec8db1 made it an atomic.Bool store because only the owner thread writes) to prove every M has observed the new generation, then flushes left-over buffers and emits an EndOfGeneration marker. Every generation is self-contained: all live goroutines get a status event (GoStatus, later GoStatusStack), all strings/stacks are in per-generation tables, and CPU samples are routed to per-generation buffers. Events are one byte of type plus LEB128 varints, decoded by a spec table (tracev2.specs). Ordering does not depend on timestamps: each G and P carries a sequence number written into events to encode a partial order (reset every generation); Ms need none because their buffers are already in order. The runtime-side API is traceAcquire/traceRelease + traceEventWriter (traceruntime.go); runtime/trace.go now carries a long invariants list (251814e580).
The parser side (internal/trace, formerly internal/trace/v2) is a streaming Reader that consumes one generation at a time, orders events with the per-G/P/M sequence numbers (order.go), and exposes typed Events with StateTransition info for Go/Proc state changes; stacks are stored as PCs with sharing (d3d390984b, ~400 MiB saved on a busy Staticcheck trace). It also reads the older formats: Go 1.11–1.19 traces are converted on the fly using Gotraceui’s optimized parser (6133c1e4e2).
Supporting data structures: traceMap was rewritten as an append-only concurrent hash-trie (5bba5b256c) because the fixed bucket table “falls over after a few thousand elements”; its traceRegionAlloc got a lock-free fast path. The tracer was made reentrant by making event writing atomic with respect to stack growth, via a small nosplit core (e76353d5a9), which allowed tracing from anywhere in the runtime. Stack capture uses frame-pointer unwinding (separate thread below).
Timeline.
- Go 1.18 — Done: v1 tracer maintenance: guard against negative/huge tick frequency on riscv64 (
a78532a412), ensure at least one tick between events (ee92daae25). Left open: nothing noted (redesign had not started in this log). - Go 1.19 — Done: user-visible CPU profile samples are emitted into the execution trace when both profilers are on, rendered as instant events in
go tool trace; samples from threads known to the runtime, including g0 and no-P contexts (2c0a9884e0);internal/tracegets Go 1.19 test data (bd4753905d). Left open: samples are best-effort (see 1.22 fix). - Go 1.20 — Done: v1 trace lock hygiene as preparation for scalability:
trace.lockonly on the system stack (6c2e327e35,42794f3871), reader parking no longer undertrace.lock(9923162f1c), stack table dumped directly into the trace buffer (d4a3466519,0293a1f308,b648591b70);tracing.enabledmade atomic (5e20f2e4df); extra-M (cgo callback) goroutines get syntheticGoCreate/GoInSyscallevents atStartTraceso the parser stops reporting “no consistent ordering” (fb6c210dc3, after a revert8adc63b3eb). Left open: “No consistent ordering” and the global lock remain structural problems of v1. - Go 1.21 — Done: user-visible STW events emitted for all pauses with kinds, not just GC (
b1aadd034c; bumps trace version to 1.21 and adds test data); trace clock formalized (b60db8f7d9); a long series of preparatory refactors for the redesign: per-M trace state in a type (2f35a655e7),traceEnabled/traceShuttingDownhelpers (3e9876cd3a,62bf7a4809),tracedSyscallEnterreplacing the subtlesysBlockTracedflag (865179164e), non-reentranttrace.lock(94a948d1b1),HeapGoalrecorded at start (192814985d); regular unwinding for cgo callbacks viam.incgocallback(3dd221a94d). Left open: still one global buffer design; redesign proposal in #60773. - Go 1.22 — Done: user-visible v2 tracer and parser added behind
GOEXPERIMENT=exectracer2(43ffe2a892); enabled by default (41f58b2295, Nov 2023);go tool tracere-implemented on the new parser:cmd/trace/v2with goroutine/task/region/MMU pages, pprof endpoints and thread/proc views (748952f494,48a6362dff,d1dcffdb4a,971f59399fand ~10 siblings); trace analyses ported (9d914015f8MutatorUtilizationV2,3b303fa981goroutine summaries,816ee0be38tasks); cgo-callback thread-ID reuse modelled correctly (6d7b3c8cd1);ProcStealfromentersyscall_gcwait(b6b72c775a); lock-hold fixes in stack table dump andreadTrace0(5249947f11,e4ea6283e4); CPU samples now written per generation using a second CPU-profile buffer chosen by the SIGPROF handler under the thread’s write flag, withtraceCPUFlushdraining the old one (f5e475edaf1.23, listed in 1.23 log). Left open: v1 still in tree; deadlock debugger disabled (74993bffe1); Windows tests disabled temporarily (505dff4fe2); goroutines blocked all generation have no stack (see 1.23). - Go 1.23 — Done: v1 tracer removed and v2 becomes the only one (
2c5849dc40, −3415 lines; flag removed55e4097cba); old-format reader (6133c1e4e2) and PC-based stacks (d3d390984b);GoStatusStackevent gives stacks to goroutines blocked for a whole generation (5879bf7e38); coroutine eventsGoCreateBlocked,GoSwitch,GoSwitchDestroyforiter.Pull, trace version 1.23 (c9c88d73f5); rewritten scalabletraceMap(5bba5b256c); user-visible experimental-event mechanism andGODEBUG=traceallocfreewhich adds 9 expensive events for heap/span/stack alloc and free plus a type-table batch (724bab1505,197101a84a,efe7a1f5d3,b6fa505c4a);traceAdvancebefore exit (20f4b6d8f9); package moved tointernal/trace(5890b023a5,664088b898). Attempted push-down ofsystemstackrequirement reverted for breaking longtest (5ec7395afc,9f13665088). Left open: tracer not reentrant; alloc/free events are experimental. - Go 1.24 — Done: tracer made reentrant via
nosplitcore (e76353d5a9); experimental batches reusable (2a3e2e9b29); deterministic reader output (1c4acea03d);Stack.Framesasiter.Seq(89a5a60da6,6eb40d158a); trace tools merged intocmd/trace(5f073d361f); event size reduced by packing G statuses (705f9848ef). Left open: none noted. - Go 1.25 — Done: user-visible
runtime/traceflight recorder (83df0afc4e, see thread below);tracev2package split out with spec validation tests and an experiments guide (0f62125487,34e8541d24,fadfe2fc80,b5f34aa4ab); trace v1 reader renamed (8028731648); sync batch withEvClockSnapshot(trace, mono and wall clock) at the start of each generation, format bumped to 1.25 (112c23612f,2d216141a1); stricter per-version event checks (715754ba86); sync event emitted even when the next generation is broken (283296195b,85f8e240fe);readTraceno longer spins on shutdown (816199e421). Left open: truncated-versus-broken traces cannot be told apart (fixed in 1.26). - Go 1.26 — Done: in-band end-of-generation event, giving unambiguous truncated trace detection and removing batch spilling (
4a7fde922f,77f911e31c); trace-writing fast path: seqlock replaced by write flag (d68aec8db1),m.profStackused instead of zeroedpcBuf(7fbd141de5, trace stacks now obeyprofstackdepth), no unique-string write for empty strings (0bc192368a); reentrancy generation-skew fix (dc9a3e2a65); P-clear vs.GOMAXPROCSrace (1de9585be2);_Psyscalleliminated with tracer shielded from_Grunning-without-P (7244e9221f);Event.Stringimprovements (de2d741667,b80ffb64d8);cmd/traceraw/text converters (bf58e7845e). Left open: alloc/free experiment tests skipped by default (4837bcc92c). - Go 1.27 — Done: no write barriers in
traceRegionAlloc(04772f022a); trace strings truncated before map insertion, fixing a fatal “alloc too large” with long region names (1cfd0dc9a3);ProcSyscallAbandonedstring (261d489139); preallocated PC array (34a03bed3c). Left open: nothing noted. - Go 1.28 — Done: test-only deflakes (
a300e279c5). Left open: nothing noted.
Frame-pointer unwinding in tracing and profiling; profile stack depth
Span: Go 1.21 → Go 1.27 · Status at HEAD: FP unwinding used by tracer, block and mutex profilers; stack depth 128 default (GODEBUG=profstackdepth); tracefpunwindoff documents which profilers it governs
Design. The tracer walks rbp chains directly (fpTracebackPCs) instead of running the generic unwinder and tracebackPCs, which makes stack capture roughly an order of magnitude cheaper and is the main reason tracing overhead became low. Inline expansion and frame skipping are deferred to report time (fpunwindExpand, refactored to use a provided buffer in b5bfb5a3ce). Where C frames without frame pointers could be on the stack (cgo callback threads), the tracer falls back to the regular unwinder so a registered cgo symbolizer can be used (3dd221a94d). Later, other profilers adopted the same machinery; for the mutex profile deferring skip/inline expansion caused duplicate records with identical stacks, so skipping is now done at record time with full inline expansion for the skipped prefix (87abb4afb6).
Timeline.
- Go 1.21 — Done: new
tracebackPCsAPI built onunwinder(3e360b035f, skip arg later removedc3db9af3a6); FP unwinding for tracing on amd64 (enabled, turned off, then re-enabled by default for amd64 and arm64 in2d83b646d6, after crash fixes and arm64 frame-pointer fixup on stack copy); disabled on plan9 (bce7aec3cd). Left open: cgo-callback threads fall back to slow path. - Go 1.23 — Done: FP unwinding for block and mutex profilers (
f4494522dc); user-visible profile stack depth raised 32 → 128 for alloc/mutex/block/threadcreate/goroutine profiles using newinternal/profilerecordtypes with unbounded stacks (1b9dc3e178),GODEBUG=profstackdepth(up to 1024, back to 32) (66cc2b7ca7), regression fix (47187a4f4f); frame pointer tracked while in syscall (2b82a4f488); mutex profile record dedup (183a40db6d, reverted6948b4df8c, relanded87abb4afb6after fixing a bounds error and inline expansion). Left open: CPU/heap profiles still use the old unwinder;MutexProfileroot-frame regression (fixed 1.24). - Go 1.24 — Done: fix lost root frames (
runtime.goexit,_LostContendedRuntimeLock) andexpandFrameout-of-bounds (c64ca8c6ef); null termination ofGoroutineProfilestacks (49e542aa85). Left open: nothing noted. - Go 1.25 — Done: account for missing frame pointer in function preamble (
742fda9524);traceStacktestable plus benchmark (75b43f9a97). Left open: nothing noted. - Go 1.27 — Done: sentinel for logical stacks in runtime-lock profile removed as dead code (
6b7206feb2). Left open: nothing noted.
Mutex and block profiles, including runtime-internal lock contention
Span: Go 1.19 → Go 1.27 · Status at HEAD: runtime-internal lock contention reported in the mutex profile with sync.Mutex-like semantics (blame the unlocker); runtimecontentionstacks setting removed
- motivating issue #57071 runtime/metrics: add a way to measure runtime-internal lock contention referenced by 2 of this thread's commits
- proposal #61015 runtime: change mutex profile to scale contention by number of blocked goroutines referenced by 1 of this thread's commits
- tracking issue #66999 runtime: correct attribution of internal mutex contention referenced by 5 of this thread's commits
Design. sync.Mutex contention is attributed to the unlocker (the critical section that delayed waiters). Runtime-internal lock2 contention was initially attributed to the waiter. The fix (4d0f955a5e, lock_spinbit.go) decides at the start of unlock2 whether to sample; if so it takes a slower unlock path, releases the lock first, then walks the list of waiting Ms (O(N)) to compute the total delay the critical section caused, and reports that with the unlocker’s call stack. Delay is measured in cputicks; zero/negative-delay events are kept (clamped to 0) instead of dropped (1416cebdd5, 433c1d3b4a: choose to drop the zero-delay event when the M’s single saved slot is contested; explicit “have event” flag). _LostContendedRuntimeLock is the placeholder frame when no stack is stored.
Timeline.
- Go 1.20 — Done: user-visible mutex profile samples scaled at record time, so changing the sample rate (even to 0) does not corrupt old samples, matching block profile (
30b1af00ff);/sync/mutex/wait/total:secondsmetric piggybacking on G runnable tracking and wait reasons (63ceff95fa). Left open: runtime-internal locks not in the profile. - Go 1.22 — Done: docs for block/mutex profiles and for runtime-internal locks in the metric (
e9eb2beeb2,3f2bf706f7); user-visible runtime-internal contention shown underGODEBUG=runtimecontentionstacks(introduced outside this bucket; testsd59bd25a30). Left open: attribution is to the waiter, not unlocker (#66999). - Go 1.23 — Done: reverts of in-flight work (
ca7d300509restoring the GODEBUG,e6b8b2f3e2); FP unwinding for the profiler (f4494522dc) and dedup fix (87abb4afb6). Left open: unlocker blame still missing. - Go 1.24 — Done: all sampled events recorded (
1416cebdd5), zero-delay events stored (433c1d3b4a), root frame fix (c64ca8c6ef). Left open: attribution (done in 1.25). - Go 1.25 — Done: verification test using
cputicksbounds comparing fast vs. slow critical sections (591c4b50c0); unlocker blame (4d0f955a5e);GODEBUG=runtimecontentionstacksremoved because the semantics now match (94c4cdc948); failing tests log the profile (35139d6e45). Left open: nothing noted. - Go 1.27 — Done: leftover
mutexeventlinkname removed (84c8483e02); sentinel cleanup (6b7206feb2); patience for trailing thread in test (6c8731962d). Left open: nothing noted. - Go 1.28 — Done: deflake (
aa194b3f68). Left open: nothing noted.
Flight recorder and trace subscribers
Span: Go 1.25 → Go 1.26 · Status at HEAD: shipped in runtime/trace (1.25); in-band generation-end signal added 1.26
- proposal #63185 runtime/trace: flight recording referenced by 1 of this thread's commits
Design. Because every generation is self-contained (see tracer thread), a circular buffer of recent generations is enough to produce a valid trace on demand. 83df0afc4e ports golang.org/x/exp/trace’s flight recorder into runtime/trace: a subscriber model lets both the ordinary tracer and the recorder consume trace data from the runtime without large runtime changes; byte readers were dropped from batch parsing; the trace clock frequency is read directly from the runtime (considered stable for process lifetime) instead of parsing it from the stream. An in-band end-of-generation signal on the internal ReadTrace (linknamed to runtime/trace) lets subscribers start/stop only at generation boundaries. Follow-ups: SetMinAge test fix (db3e02994c), orders test (c8c3d661b0), 96a6e147b2 marking linknames used by runtime/trace.
Timeline.
- Go 1.25 — Done: user-visible flight recorder (
83df0afc4e),TestSubscribersinfrastructure (f2d96272cb1.26 in 1.26 log). Left open: “Future optimizations are planned for” the multi-subscriber mechanism (commit message). - Go 1.26 — Done: in-band
EndOfGenerationforruntime.ReadTrace+ parser (4a7fde922f);Logbenchmark (8d9906cd34); test traces dumped on failure (ee4d2c312d,89614ad264). Left open: nothing noted.
CPU profiling, goroutine profiling, pprof labels and the protobuf writer
Span: Go 1.18 → Go 1.28 · Status at HEAD: maintenance; labels in tracebacks behind GODEBUG=tracebacklabels=1
Paths not taken: Background CPU profiling API (never landed) · Hardware-counter (PMU / perf_events) profiling (dormant)
Earlier history: CPU profiler and runtime/pprof writers (legacy format to compressed protobuf) (Go 1.0 → Go 1.15) · Profile labels (Go 1.9 → Go 1.15)
- proposal #76349 runtime: add runtime/pprof goroutine labels to tracebacks referenced by 1 of this thread's commits
Design. CPU samples flow SIGPROF handler → profBuf (lock-free ring with tags carrying pprof label pointers) → reader goroutine → profileBuilder (runtime/pprof/proto.go) which builds the protobuf, merging inlined fake PCs back into single locations. Label sets were map[string]string; ca63101df4 replaced them with a slice-based LabelSet-backed labelMap and an ordered-keys fast path (geomean −49% time, −52% bytes, set-one 153ns→75ns), partly to let eBPF profilers read labels without map internals. The goroutine profile snapshots every goroutine’s stack with a per-goroutine state machine (absent → in-progress → satisfied) while the world runs; 281cfcfc1b changed system goroutines (finalizer, multiple cleanup goroutines) to participate in the state machine and decide at record time whether to emit a stack.
Timeline.
- Go 1.18 — Done: missing-label fixes (
da7173a2ed,9b0de0854d; one revertedc6a0b6f2de), profile dump includes labels (90f47dbba6), first CPU record tag consumed (a3ae45ebe1), goroutine-profile traceback bypasses the scheduler for cgo traceback to avoid deadlock at exitsyscall under STW (8c73f80400), histogram bucket off-by-one fix (2e9dcb5086). Left open: labels on racecall frames in tests. - Go 1.19 — Done: user-visible MaxRSS reported on all unix platforms (
c1f22134f2); inlining correctness in the proto builder: PC reuse by inline NOPs (bd77d6e240),allFramescache bug producing wrongC<-B<-A,Alocations (ba8310cf29),pcDeckindexing (0d33a99675); finalizer G handling and race annotations in the concurrent goroutine profile (8fdd277fe6,209942fa88);profBufread-before-advance fix (1926fa5f84). Left open: nothing noted. - Go 1.20 — Done: user-visible 4-column memprofile text form forced so
pprofdoesn’t collapse alloc==inuse into 2 columns (c2ede92a0d); Windows memory mappings (28afa5b176);Function.start_line(a401468b00); Windows MaxRSS (2cc6983a21);prof.hz,profbuffields to atomic types (54cf1b107d,c3c10f5d67etc.);MemProfileRateno longer set in init (f63b268b9a); empty-stack docs (0bbd67e52f). Left open: nothing noted. - Go 1.21 — Done: generic instantiated symbol names recorded in CPU profiles so PGO can match them (
10ad6c91de); concurrent goroutine profile flag removed, making it the only path (469afbcc46); lazy regexps ininternal/profilecut init 185µs → 6µs (f53137f95c). Left open: nothing noted. - Go 1.22 — Done: darwin memory mappings via
vmmap/Mach calls (b7c826d2c4; vmmap retry fix9312f47d0a1.23 landed in 1.23); labels for the goroutine requesting a goroutine profile (9cdcb01320); profile of empty-sample protos still fully decoded (e1c0349a7c); generics names and inlined generics locations (20a03fc713,d95e25e83c);ticksPerSecondneeds 5ms instead of a 100ms thread sleep and runs on a goroutine sleep (130baf3d42). Left open: Darwin profiling needed non-blocking reads (c6ee8e31e3). - Go 1.23 — Done: legacy profile parser removed from
internal/profile(2f6a25f447); one fewer STW when reading goroutine profile config (f96d9a6432); linker no longer disables memory profiling when onlyWriteHeapProfileis reachable (82c14346d8); stack depth 128 (see FP thread). Left open: nothing noted. - Go 1.24 — Done: user-visible cheaper pprof labels (
ca63101df4, benchmark411ba0ae86); heap profiles hideinternal/runtime/mapsframes (d0c9142ce3). Left open: nothing noted. - Go 1.25 — Done: write errors from profile output propagated (
4aeb9ba0de); system goroutines join the goroutine profile state machine (281cfcfc1b);TestCgoCallbackPprofskipped where profiling is broken (7dd7d70c03). Left open: nothing noted. - Go 1.26 — Done: fix double
notewakeupwhen two writes hit a fullprofBuf(b604148c4e); remove fixed sleep in CPU profile reader (121bc3e464);GODEBUG=tracebacklabels=1adds escaped pprof labels to goroutine headers in tracebacks, with a newruntime.quotedprint type (6851795fb6);internal/profile.Parseallocs −36% (c5559344ac). Left open: escaping of non-ASCII runes and invalid UTF-8 “can be improved” (commit message); opt-in to experiment with format. - Go 1.27 — Done: memory profile records drop redundant bytes fields, 64 bytes saved per
memRecord(d247ed00e4); gzip close error returned (8b672822b2); buffered writer in pprof (6eb5a41228); dangling-ID rejection (93a285f2bc1.28 in 1.28 log). Left open: nothing noted. - Go 1.28 — Done: stop logging an error when CPU sampling rate is already set (
2482f9c242); synthetic mappings for WebAssembly profiles (412f013cf8); reject dangling location IDs ininternal/profile.Parse(93a285f2bc). Left open: nothing noted.
Goroutine leak profile (GC-based leak detection)
Span: Go 1.26 → Go 1.27 · Status at HEAD: goroutineleak profile on by default; GOEXPERIMENT=goroutineleakprofile removed
- design doc Proposal: Goroutine leak detection via garbage collection Georgian-Vlad Saioc, Milind Chabbi, 14 Aug 2025 · linked from #74609
- proposal #74609 runtime/pprof,runtime: new goroutine leak profile referenced by 4 of this thread's commits
- tracking issue #75280 runtime/pprof,runtime: add new goroutine leak profile as a GOEXPERIMENT linked from #74609
Design. Proposal #74609: using the garbage collector to find goroutines blocked forever on channels/sync primitives that no reachable goroutine can unblock. The landing change (8c68a1c1ab) touched ~94 files (runtime channel code, GC marking in mbitmap.go, net/http/pprof, linker support) and registers a new pprof profile type; with the experiment off the profile is simply not registered. Stacks are collected without the goroutine-profile semaphore (8873e8bea2: recorded inline via saveg rather than doRecordGoroutineProfile, which had side-effects on goroutineProfile fields).
Timeline.
- Go 1.26 — Done: user-visible experiment added, default off (
8c68a1c1ab); cleanup of profile writing, stress tests for concurrent leak and goroutine profile requests (8873e8bea2); noopt test fixes (625d8e9b9c). Left open: stress subtests flaky (8f45611e78). - Go 1.27 — Done: main goroutine blocked on
select{}excluded (d81ba6c35d); user-visible experiment enabled by default (253aa2a12a, 2026-05-05) and flag deleted the same day (16449179ec,afcf04cb64); flake mitigation by allowing 4–6 leaked goroutines, tracked in #79452 (320e0be23d). Left open: root cause of the 5-instead-of-6 case unknown (320e0be23d).
runtime/metrics and MemStats
Span: Go 1.18 → Go 1.27 · Status at HEAD: ~dozens of metrics; Read guaranteed not to stop the world (test)
Earlier history: Heap profile accounting and MemStats (Go 1.0 → Go 1.17) · runtime/metrics (Go 1.16 → Go 1.17)
- design doc Proposal: API for unstable runtime metrics Michael Knyszek · listed in Go: A Documentary
- proposal #15490 runtime/metrics: add goroutine state counts, total goroutines created, total threads referenced by 3 of this thread's commits
- proposal #47216 runtime/metrics: add GOMAXPROCS, NumCgoCall, GC CPU time referenced by 3 of this thread's commits
- proposal #56857 runtime/metrics: add /gc/heap/live:bytes referenced by 8 of this thread's commits
Design. Metrics are described in runtime/metrics/description.go (docs generated from metrics.All) and computed in runtime/metrics.go from metricsSema-protected aggregates (heapStatsAggregate, sysStatsAggregate, gcStatsAggregate, cpuStatsAggregate) with dependency tracking (statDep). Time histograms (timeHistogram) use log-linear buckets: the off-by-one that doubled sub-bucket distance was fixed with a simpler formula (2e9dcb5086), then buckets were coarsened with a minimum bucket bit and fewer sub-buckets for ~3x smaller histograms (87eda2a782). Counters in P-local stats use statsSeq (atomic type, 2883af63c2), consistentHeapStats acquire/release made nosplit (df1837799d).
Timeline.
- Go 1.19 — Done: user-visible
/sched/gomaxprocs:threads(ece6ac4d4d),/cgo/go-to-c-calls:calls(9bd6e2776f), last GC cycle with limiter enabled (cfccb5cb7c); total allocation stats stored as uint64 (7c404d59db); user-visibledebug.SetMemoryLimitexported (f01c20bf2b, GC area). Left open: nothing noted. - Go 1.20 — Done: user-visible estimated CPU time breakdown metrics (
/cpu/classes/...) with total CPU (b7c28f484d);/sync/mutex/wait/total:seconds(63ceff95fa); timeHistogram atomics and size cut (b04e4637db,87eda2a782). Left open: CPU numbers are estimates, not on-CPU counters. - Go 1.21 — Done: user-visible
/gc/heap/live:bytes(ef2bb813c8),/gc/gogc:percent(1eae5c160d),/gc/gomemlimit:bytes(2544b1051e),/gc/scan/{stack,globals,heap,total}:bytes(8b51eb5199,a63afe4720,131267f147,5124371c1c; consistency fix72bf8acee9),/godebug/non-default-behavior/*counters (213495a4a6);/sched/latencies:secondsmarked cumulative (c2c787d73e); doc clarifications of OS stack metrics (b2215e49c7);ReadMemStatsnil check before stack switch (e126572f8a). Left open:/gc/scan/*was always 0 (fixed in 1.22). - Go 1.22 — Done: user-visible
/sched/pauses/{stopping,total}/{gc,other}:seconds;/gc/pauses:secondsdeprecated as an alias for the new gc/total (6ef98ac87c);/gc/scan/*fixed (2186909d86);ReadMetricsSlowavoids stack movement (6a32ecc059); ReadMemStats debug assertions behind double-check (b2efd1de97). Left open: nothing noted. - Go 1.23 — Done: GC pause CPU stats updated inside the STW to remove a race, tests tightened by 10x (
017478a963); usesmaxprocsnotstwprocs(ae0a08dee9); IncNonDefault coverage test (74726defe9,418e6d559e,20b79fd577). Left open: nothing noted. - Go 1.25 — Done: ptr/scalar bitmap metric removed (
60d3bcdec3);HeapGoalreported 0 when GC disabled (1947c4233a). Left open: nothing noted. - Go 1.26 — Done: user-visible goroutine scheduling-state metrics (
13df972f68), total goroutines created (ab8121a407), current thread count (78a3968c2c), cleanup/finalizer queue metrics (a4d99770c0). Left open: nothing noted. - Go 1.27 — Done: panic in
Readwith empty slice fixed (6aef900af4); test ensuringReadnever needs a stop-the-world (24ec7d2120). Left open: nothing noted.
Coverage redesign: runtime/coverage, internal/coverage, covdata
Span: Go 1.20 → Go 1.24 · Status at HEAD: default since 1.20; coverageredesign experiment removed (41298239cf, Jan 2025)
- design doc Proposal: extend code coverage testing to include applications Than McIntosh, 2022-03-02 · golang/proposal
- proposal #51430 cmd/cover: extend coverage testing to include applications referenced by 11 of this thread's commits
Design. Instead of one global counter file written by testing, cmd/cover instruments each package with a per-package counter array and a meta-data blob (function units, strings, 128-bit hash). At init, instrumented packages call a registration hook (runtime/covermeta.go; fixed package IDs for runtime, internal/cpu etc. whose functions run before init, cf83a490e4). On exit a runtime exit hook (07bdf1dc54: addExitHook(f, runOnNonZeroExit), run in reverse registration order) writes a meta-data file and a counter-data file into GOCOVERDIR (87db4ffada). Encoders/decoders and a streaming “pod” reader live in internal/coverage (f951f697c4, 84f95aa811, efa3f1749f, 7a74829858); go tool covdata supports merge, intersect, subtract, percent, pkglist, debugdump, textfmt. User APIs runtime/coverage.WriteMeta/WriteCounters/ClearCounters allow capture from long-running programs (ecd112c0d1), restricted to atomic mode because of relaxed-memory-model tearing seen on ppc64 (61e5ea4929).
Timeline.
- Go 1.20 — Done: user-visible full pipeline above: encoder/decoder (
f951f697c4,84f95aa811), exit hook (07bdf1dc54), runtime emit (87db4ffada), covdata tool (7a74829858), counter I/O (efa3f1749f), user APIs (ecd112c0d1), package registration hook (cf83a490e4); experimentGOEXPERIMENT=coverageredesignadded (a3434b71a3) and enabled by default (9910f8a403, 2022-09-29); atomic counter reads (cddf792428);-coverprofilelocal-package path capture restored (2730170f63);package mainhandling fix (7973b0e508). Left open: covdata mode clash policy (relaxed in 1.21a371fa5e701.21). - Go 1.21 — Done: user-visible atomic-only counter APIs (
61e5ea4929); count-vs-emit discrepancy fix (39957b5d89);coverage.snapshotfortesting.Coverage()(ef67022471); auxiliary meta-data files for packages without tests (c99d966c17);EmitPercentaggregation (380529d5c3); counter file ordering (0f91f92ee0). Left open: go command wiring for no-test packages (landed in 1.22). - Go 1.22 — Done: user-visible
cmd/cover“emit meta file” mode and go command handling for packages without tests (e356aa656d,d1cb5c0605);covcmddefs moved tocmd(3fb1d95149,98bacdeafe); compilerFixupmerge (3378adc517). Left open: nothing noted. - Go 1.23 — Done:
go:linknameuses removed (180ea45566,df4e49366c,cd41d71785);-coverpkgpercentage fix (a85b8810c4,7196db9e66). Left open: nothing noted. - Go 1.24 — Done: 128-bit FNV-1a replaces MD5 for file identity (
092d18b318); text-format output with multiple packages fixed (50455385b0,40b3c0e58a); stringslite added to the hard-coded runtime package list (966688a43a). Left open: nothing noted. - Go 1.27 — Done: first write error is now recorded (
78314e0140). Left open: nothing noted.
Race detector, ASan, MSan and Valgrind integration
Span: Go 1.18 → Go 1.28 · Status at HEAD: race on linux/riscv64, loong64, s390x; asan/msan linux incl. loong64, riscv64, ppc64le; build-tag Valgrind support
Earlier history: Race detector integration (Go 1.1 → Go 1.17) · MSan and sanitizer integration (Go 1.6 → Go 1.16)
- proposal #44853 runtime: enable address sanitizer in Go referenced by 2 of this thread's commits
- discussion Enable address sanitizer in Go golang-dev · linked from #44853
- tracking issue #64611 all: add internal/asan and internal/msan packages referenced by 3 of this thread's commits
Design. Each sanitizer needs the runtime to tell it about Go memory. The race detector uses prebuilt runtime/race/*.syso TSAN runtimes and ABI-correct racecall trampolines per arch; ASan wrappers asanread/asanwrite (NOSPLIT) are inserted by the compiler, and the runtime poisons/unpoisons allocations; MSan analogously. Valgrind (40b19b56a9) is gated by a build tag and implemented without cgo or Valgrind headers: a tiny assembly function emits the magic client-request instruction sequence. Memory is modeled as a two-level mempool: heap arenas are a pool created with VALGRIND_CREATE_MEMPOOL_EXT (METAPOOL and AUTO_FREE), spans are “superblocks” (MEMPOOL_ALLOC), and objects are MALLOCLIKE_BLOCK; stacks are annotated too.
Timeline.
- Go 1.18 — Done: ASan runtime support,
asanread/asanwrite,runtime/asanpackage (adfb85b315, NOSPLIT2c58bb2e42, C declarationsa0f373ca08, error location fix91e782106e); race functions ABIInternal (40fce515f9,d93cc8cb96), arm64 stack space (d27248c52f). Left open: globals not yet checked. - Go 1.19 — Done: user-visible new TSAN runtime v3: 2x smaller shadow memory, vectorized race detection, fixed 512-byte vector clocks, unlimited goroutines; e.g. encoding/json CodeDecoder −56%, net/http ClientServerParallel4 −92%,
syncshort tests −82% (d6a1ffd624, syso995a604591); race on s390x (87cf92e7d5, syso7c680974c6), PPC64LE C ABI (c8110c3df6); ASan on riscv64 (9956996f6e); ASan global checks attempted and reverted twice (1e5987635c,507b70d1e8,265b5fd2f1); amd64 syso avoids SSE4 (846490110a). Left open: ASan instrumentation of globals (reverted for breaking cmd/go tests). - Go 1.20 — Done: ASan on ppc64le (
9e810997c0); race syso subarch versioning and GOAMD64=v3 image (e665de2861,c75b10be0b,18baca6765); no cgo needed for race on macOS (5bf9aeba1c). Left open: nothing noted. - Go 1.21 — Done: race tracebacks symbolize wrappers last (
cb6b45a993);__tsan_finimarks entering non-Go code (30886390c2); 1-byte load for address checking inracecallatomic(1a7709d6af); ppc64le thunk cleanup (e78871ea07). Left open: nothing noted. - Go 1.22 — Done: ASan on linux/loong64 (
e9c14a7780); race syso with atomic And/Or (6b43a172d8); testing attributes races to subtests via monotonic counts (55b8e16b2e); asan asm fix on amd64 (d9f7e1dc73). Left open: nothing noted. - Go 1.23 — Done:
internal/asanandinternal/msanpackages for use from std (db99b9b17c,e1e466f80c,8f7df2256e); wakeableSleep modeled in the race detector (0b12e3d81c); arm64 racecall saved-FP clobber fixed (234390e741); user-visible publicsync/atomicAnd/Or with race instrumentation (192d65e46b). Left open: nothing noted. - Go 1.24 — Done:
internal/racecontents linknamed into runtime (e25b913127); race enabled forinternal/runtime/maps(3a6795554d); concurrent map access fatal error treated as a race hit in tests (fc5e8f2f6b). Left open: nothing noted. - Go 1.25 — Done: race on loong64 (
bce8486ebe); gsignal checks inasancall/msancall/racecall(11c86ddcb8); user-visible ASan builds run__lsan_do_leak_checkat exit (645ea53019) and unregister root regions on free (56e5476e10); msan memmove through cgo (969a0da362); precise Go data/bss bounds for race (c1fc209c41); user-visible Valgrind instrumentation (40b19b56a9); GDB minimum version raised to 10 (8d767ff38d). Left open: Valgrind--leak-check=yesreports errors because “likely need more annotations”; async preemption and GC very slow under Valgrind (commit message). - Go 1.26 — Done: race on linux/riscv64 (
8bcda6c79d); msan uses a dedicated copy routine (5f4b5f1a19); race darwin syso refreshed/restored (8282b72d62,1b263fc604); ppc64x atomic And/Or wrappers (44cb82449e). Left open: nothing noted. - Go 1.27 — Done: LLVM zero-initialization fix (
325eedb2a9); arm64 syso without outline atomics (bcdf61fef4); test harness tolerates WaitGroup-reuse panic (86935037f41.28). Left open: nothing noted. - Go 1.28 — Done: loong64 MSan label spelling fix (
45c2e35625). Left open: nothing noted.
Crash output, runtime/debug, BuildInfo and debuglog
Span: Go 1.18 → Go 1.26 · Status at HEAD: debug.SetCrashOutput public; build info includes VCS and settings; debuglog compiled in tests
Earlier history: GODEBUG, runtime/debug, debug log and traceback knobs (Go 1.0 → Go 1.17) · Traceback correctness and inlining (diagnostic output) (Go 1.0 → Go 1.17) · GDB runtime support (Go 1.0 → Go 1.15)
- proposal #42888 runtime/debug: add SetCrashOutput(file *os.File) referenced by 4 of this thread's commits
- proposal #37475 cmd/go: stamp git/vcs current HEAD hash/commit hash/dirty bit in binaries referenced by 3 of this thread's commits
- proposal #67182 runtime/debug: add SetCrashOutput options referenced by 2 of this thread's commits
Design. debug.SetCrashOutput(f, CrashOptions) (1bb947b2eb) makes the runtime write fatal panic output also to a file descriptor (crashFD initialized to -1, guarded against FD changes during panic 638b9023e3), so a watchdog child process can record unhandled panics for telemetry. debuglog (runtime/debuglog.go) is a per-M lock-free ring log; originally the debuglog build tag made the logger type-erased through a const. 548158c4a5 makes the tag select a type alias to dloggerImpl or dloggerFake, so the implementation is always compiled and tested (runtime.a +0.9%, no cmd/go size change).
Timeline.
- Go 1.18 — Done: user-visible
runtime/debug.BuildInfogainsGoVersion(85a068fdf2),debug/buildinfopackage (434cdd0337), VCS stamping (a37bebc042),Settingsaligned with flags/env (931d80ec17),Parse/StringreplaceMarshal(9cec77ac11). Left open: nothing noted. - Go 1.19 — Done: debuglog avoids string allocation (
73587b71a6). Left open: nothing noted. - Go 1.20 — Done: docs and debuglog
unsafe.Sliceavoided (dadd80ae20,c45ebef05e). Left open: nothing noted. - Go 1.23 — Done: user-visible
SetCrashOutput(1bb947b2eb, examplef0a862ee4d, dup semanticsb27d02c07b,CrashOptionsparam3f7a030e02, variadic removeda22cb5cabe). Left open: “placeholder for future options (e.g. JSON)”. - Go 1.24 — Done: debuglog type-toggled (
548158c4a5,eb6743d9d7), tests always run (3e1bda08fb,13b3af0391,c7227bc72cforces >=10 shards),ParseBuildInfo/Stringdocs (0c460ad014). Left open: nothing noted. - Go 1.25 — Done:
DefaultGODEBUGdocumented as a build setting (c4136a433c);SetCrashOutputexample no longer passes parent env (3672a09a48). Left open: nothing noted. - Go 1.26 — Done:
dlog.hexdump(8c41a482f9);tracebacklabels(6851795fb6, see profiling thread). Left open: nothing noted.
Changes in this slice that are not part of a thread are listed on the release pages: 1.18 · 1.19 · 1.20 · 1.21 · 1.22 · 1.23 · 1.24 · 1.25 · 1.26 · 1.27