Chapter 14 Execution Stack Management on Go: Under the Hood

14.1 Design of Contiguous Stacks

Mon, 01 Jan 0001 00:00:00 +0000

14.1 Design of Contiguous Stacks

Every goroutine carries an execution stack. It holds the local variables, arguments, and return addresses of function calls, and it is the physical medium that lets a program be “currently executing”. In the C programs the reader is familiar with, this stack is provided by the operating system thread, its size fixed once at thread creation (ulimit -s defaults are often 8MB) and unchanged thereafter. Go took a different road: a goroutine’s stack is not a thread stack but a segment of memory that the runtime allocates on the heap, manages itself, and can resize, starting at just 2KB.

14.2 Stack Allocation and Caching

Mon, 01 Jan 0001 00:00:00 +0000

14.2 Stack Allocation and Caching

14.1 located the execution stack as a contiguous range of memory [lo, hi): it is managed by the runtime and ultimately lives on the heap, so the moment a Goroutine starts, someone has to carve out this range of addresses for it. The questions follow immediately: who carves it? how large? and once Goroutines are created and destroyed over and over, if these two- or three-kilobyte chunks of memory had to be requested directly from the operating system or the global heap every time, the cost of locks and system calls would crush the scheduler at once.

14.3 Stack Growth

Mon, 01 Jan 0001 00:00:00 +0000

14.3 Stack Growth

14.1 laid out the reasons Go chose contiguous stacks: a stack starts at just 2KB, and on overflow it swaps in a new stack twice as large and copies the entire old stack across. This section answers only one of the questions involved: how the runtime knows, at the “right moment,” that a stack is about to fill up, and how it hands control all the way over to the code responsible for the growth. The details of the stack copy itself are left to 14.4.

14.4 Stack Copying and Pointer Adjustment

Mon, 01 Jan 0001 00:00:00 +0000

14.4 Stack Copying and Pointer Adjustment

The cost of contiguous stacks is concentrated in one place: when a goroutine’s stack overflows, the runtime allocates a new stack twice the size, moves the entire old stack over, and then frees the old stack. The moving step looks like nothing more than a single memmove, copying the bytes of [old.lo, old.hi) into [new.lo, new.hi). If it really were just copying bytes, a problem would arise: variables on the stack may hold pointers into the stack itself, for example a local variable that has taken the address of another local. After the bytes are moved verbatim to a new address, the values of these pointers are unchanged, still pointing at locations in the old stack, and the old stack is about to be reclaimed. The moment the copy finishes, they become dangling pointers.

14.5 Stack Shrinking and Evolution

Mon, 01 Jan 0001 00:00:00 +0000

14.5 Stack Shrinking and Evolution

14.4 was about the stack “growing up”: when a function prologue finds the stack nearly exhausted, the runtime copies the whole stack onto a larger block of memory, and the stack pointer shifts along with it. This is a one-directional force driven by execution itself, and it only ever makes the stack larger. Yet a goroutine’s stack depth is not monotonic. A recursion may descend very deep, then return layer by layer back to a shallow point; the stack it actually needs has long since shrunk back, but the large stack expanded for that deep recursion is still held in its name. With growth but no return, every goroutine’s stack would eventually settle at its historical deepest point. For a program that may hold millions of goroutines, this bill is unbearable.