Summary

The provided web content discusses the "Stop the World" (STW) mechanism in Go's garbage collector, detailing how it operates, potential latency issues, and how system calls are handled during STW phases.

Abstract

In Go's garbage collection process, "Stop the World" (STW) is a critical phase where the program's execution is suspended to allow the garbage collector to perform memory root scanning and add write barriers. This phase involves preempting all running goroutines, marking processors (P) as stopped, detaching each machine (M) from P, and moving Ms to the idle list, while goroutines wait in the global queue. The STW phase can also impact system calls, as they may return while the world is stopped, leading to goroutines being queued until the world resumes. The article highlights that while STW is designed to be quick, certain scenarios, such as goroutines with long-running loops without function calls, can cause significant latencies. However, the article notes that Go 1.14 introduced asynchronous preemption to mitigate this issue.

Opinions

The author suggests that understanding the internal workings of STW is important for Go developers.
There is an acknowledgment that STW phases can cause noticeable latencies in specific scenarios, which can be problematic for certain applications.
The article implies that the introduction of asynchronous preemption in Go 1.14 is a significant improvement in handling STW-related latencies.
The author recommends reading additional resources provided in the article for a deeper understanding of the garbage collector cycle and asynchronous preemption.
The use of visual aids (illustrations and tracing graphs) is advocated as a means to better comprehend the STW process and its impact on system calls and latencies.

Go: How Does Go Stop the World?

Illustration created for “A Journey With Go”, made from the original Go Gopher, created by Renee French.

ℹ️ This article is based on Go 1.13.

A “Stop the World” (STW) is a crucial phase in some garbage collector algorithms to get track of the memory. It suspends the execution of the program to scan the memory roots and add write barriers. Let’s review how it works internally and the potential issues it could face.

Stop the World

Stopping the program means stopping the running goroutines. Here is a simple program that will “Stop the World”:

func main() {
   runtime.GC()
}

Running the garbage collector will trigger two “Stop the World” phases.

For more information about the garbage collector cycle, I suggest you read my article “Go: How Does the Garbage Collector Mark the Memory?”

The first step of this phase is to preempt all running goroutines:

Once the goroutines are preempted, they will be stopped at a safe point. Meanwhile, the processors P — running code or in the idle list — will be marked as stopped to not be used to run any code:

Then, the Go scheduler will run and detach each M from their respective P and put them in the idle lists:

Regarding the goroutines running on each M, they will wait in the global queue:

Goroutines are waiting in the global queue

Then, once the world is stopped, the only active goroutine can safely run and start the world when the work is done. The tracing will help to see when this phase happens:

Stop the World “STW” phase in the tracing

System calls

The phase “Stop the World” could impact system calls as well since they could return while the world is stopped. Let’s take an example with a program that intensively does system calls and see how it is handled:

func main() {
   var wg sync.WaitGroup
   wg.Add(10)

   for i := 0; i < 10; i++ {
      go func() {
         http.Get(`https://httpstat.us/200`)
         wg.Done()
      }()
   }

   wg.Wait()
}

Here is the tracing:

The system call here is exiting while the world is stopped. However, since there is no available P — they are all marked as stopped, as seen in the previous section — the goroutine will be put in the global queue and will run later when the world resumes.

Latencies

The third step of the “Stop the World” involves the M’s to be all detached from their P. However, Go will wait for them to stop voluntarily: when the scheduler runs, during syscall, etc. Waiting for a goroutine to be preempted should be fast, but in some cases, it could lead to some latencies. Let’s take an example that will show an extreme case:

func main() {
   var wg sync.WaitGroup

   var t int
   for i := 0;i < 20 ;i++  {
      wg.Add(1)
      go func() {
         for i := 0;i < 1000000000 ;i++ {
            t++
         }
         wg.Done()
      }()
   }
   wg.Wait()

   runtime.GC()
}

Here, the “Stop the World” phase takes 2.6 seconds:

A goroutine without function calls will not be preempted, and its P will not be released before the end of the task. That will force the “Stop the World” to wait for it.

ℹ️ The issue raised in this section is now fixed in Go 1.14 with the asynchronous preemption. Indeed, Go is now able to send signals to each P to preempt the running goroutine. For more details about asynchronous preemption, I suggest you read “Go: Asynchronous Preemption.”