Source file src/runtime/mbarrier.go
1 // Copyright 2015 The Go Authors. All rights reserved. 2 // Use of this source code is governed by a BSD-style 3 // license that can be found in the LICENSE file. 4 5 // Garbage collector: write barriers. 6 // 7 // For the concurrent garbage collector, the Go compiler implements 8 // updates to pointer-valued fields that may be in heap objects by 9 // emitting calls to write barriers. The main write barrier for 10 // individual pointer writes is gcWriteBarrier and is implemented in 11 // assembly. This file contains write barrier entry points for bulk 12 // operations. See also mwbbuf.go. 13 14 package runtime 15 16 import ( 17 "internal/abi" 18 "internal/goarch" 19 "internal/goexperiment" 20 "unsafe" 21 ) 22 23 // Go uses a hybrid barrier that combines a Yuasa-style deletion 24 // barrier—which shades the object whose reference is being 25 // overwritten—with Dijkstra insertion barrier—which shades the object 26 // whose reference is being written. The insertion part of the barrier 27 // is necessary while the calling goroutine's stack is grey. In 28 // pseudocode, the barrier is: 29 // 30 // writePointer(slot, ptr): 31 // shade(*slot) 32 // if current stack is grey: 33 // shade(ptr) 34 // *slot = ptr 35 // 36 // slot is the destination in Go code. 37 // ptr is the value that goes into the slot in Go code. 38 // 39 // Shade indicates that it has seen a white pointer by adding the referent 40 // to wbuf as well as marking it. 41 // 42 // The two shades and the condition work together to prevent a mutator 43 // from hiding an object from the garbage collector: 44 // 45 // 1. shade(*slot) prevents a mutator from hiding an object by moving 46 // the sole pointer to it from the heap to its stack. If it attempts 47 // to unlink an object from the heap, this will shade it. 48 // 49 // 2. shade(ptr) prevents a mutator from hiding an object by moving 50 // the sole pointer to it from its stack into a black object in the 51 // heap. If it attempts to install the pointer into a black object, 52 // this will shade it. 53 // 54 // 3. Once a goroutine's stack is black, the shade(ptr) becomes 55 // unnecessary. shade(ptr) prevents hiding an object by moving it from 56 // the stack to the heap, but this requires first having a pointer 57 // hidden on the stack. Immediately after a stack is scanned, it only 58 // points to shaded objects, so it's not hiding anything, and the 59 // shade(*slot) prevents it from hiding any other pointers on its 60 // stack. 61 // 62 // For a detailed description of this barrier and proof of 63 // correctness, see https://github.com/golang/proposal/blob/master/design/17503-eliminate-rescan.md 64 // 65 // 66 // 67 // Dealing with memory ordering: 68 // 69 // Both the Yuasa and Dijkstra barriers can be made conditional on the 70 // color of the object containing the slot. We chose not to make these 71 // conditional because the cost of ensuring that the object holding 72 // the slot doesn't concurrently change color without the mutator 73 // noticing seems prohibitive. 74 // 75 // Consider the following example where the mutator writes into 76 // a slot and then loads the slot's mark bit while the GC thread 77 // writes to the slot's mark bit and then as part of scanning reads 78 // the slot. 79 // 80 // Initially both [slot] and [slotmark] are 0 (nil) 81 // Mutator thread GC thread 82 // st [slot], ptr st [slotmark], 1 83 // 84 // ld r1, [slotmark] ld r2, [slot] 85 // 86 // Without an expensive memory barrier between the st and the ld, the final 87 // result on most HW (including 386/amd64) can be r1==r2==0. This is a classic 88 // example of what can happen when loads are allowed to be reordered with older 89 // stores (avoiding such reorderings lies at the heart of the classic 90 // Peterson/Dekker algorithms for mutual exclusion). Rather than require memory 91 // barriers, which will slow down both the mutator and the GC, we always grey 92 // the ptr object regardless of the slot's color. 93 // 94 // Another place where we intentionally omit memory barriers is when 95 // accessing mheap_.arena_used to check if a pointer points into the 96 // heap. On relaxed memory machines, it's possible for a mutator to 97 // extend the size of the heap by updating arena_used, allocate an 98 // object from this new region, and publish a pointer to that object, 99 // but for tracing running on another processor to observe the pointer 100 // but use the old value of arena_used. In this case, tracing will not 101 // mark the object, even though it's reachable. However, the mutator 102 // is guaranteed to execute a write barrier when it publishes the 103 // pointer, so it will take care of marking the object. A general 104 // consequence of this is that the garbage collector may cache the 105 // value of mheap_.arena_used. (See issue #9984.) 106 // 107 // 108 // Stack writes: 109 // 110 // The compiler omits write barriers for writes to the current frame, 111 // but if a stack pointer has been passed down the call stack, the 112 // compiler will generate a write barrier for writes through that 113 // pointer (because it doesn't know it's not a heap pointer). 114 // 115 // 116 // Global writes: 117 // 118 // The Go garbage collector requires write barriers when heap pointers 119 // are stored in globals. Many garbage collectors ignore writes to 120 // globals and instead pick up global -> heap pointers during 121 // termination. This increases pause time, so we instead rely on write 122 // barriers for writes to globals so that we don't have to rescan 123 // global during mark termination. 124 // 125 // 126 // Publication ordering: 127 // 128 // The write barrier is *pre-publication*, meaning that the write 129 // barrier happens prior to the *slot = ptr write that may make ptr 130 // reachable by some goroutine that currently cannot reach it. 131 // 132 // 133 // Signal handler pointer writes: 134 // 135 // In general, the signal handler cannot safely invoke the write 136 // barrier because it may run without a P or even during the write 137 // barrier. 138 // 139 // There is exactly one exception: profbuf.go omits a barrier during 140 // signal handler profile logging. That's safe only because of the 141 // deletion barrier. See profbuf.go for a detailed argument. If we 142 // remove the deletion barrier, we'll have to work out a new way to 143 // handle the profile logging. 144 145 // typedmemmove copies a value of type typ to dst from src. 146 // Must be nosplit, see #16026. 147 // 148 // TODO: Perfect for go:nosplitrec since we can't have a safe point 149 // anywhere in the bulk barrier or memmove. 150 // 151 // typedmemmove should be an internal detail, 152 // but widely used packages access it using linkname. 153 // Notable members of the hall of shame include: 154 // - github.com/RomiChan/protobuf 155 // - github.com/segmentio/encoding 156 // 157 // Do not remove or change the type signature. 158 // See go.dev/issue/67401. 159 // 160 //go:linkname typedmemmove 161 //go:nosplit 162 func typedmemmove(typ *abi.Type, dst, src unsafe.Pointer) { 163 if dst == src { 164 return 165 } 166 if writeBarrier.enabled && typ.Pointers() { 167 // This always copies a full value of type typ so it's safe 168 // to pass typ along as an optimization. See the comment on 169 // bulkBarrierPreWrite. 170 bulkBarrierPreWrite(uintptr(dst), uintptr(src), typ.PtrBytes, typ) 171 } 172 // There's a race here: if some other goroutine can write to 173 // src, it may change some pointer in src after we've 174 // performed the write barrier but before we perform the 175 // memory copy. This safe because the write performed by that 176 // other goroutine must also be accompanied by a write 177 // barrier, so at worst we've unnecessarily greyed the old 178 // pointer that was in src. 179 memmove(dst, src, typ.Size_) 180 if goexperiment.CgoCheck2 { 181 cgoCheckMemmove2(typ, dst, src, 0, typ.Size_) 182 } 183 } 184 185 // wbZero performs the write barrier operations necessary before 186 // zeroing a region of memory at address dst of type typ. 187 // Does not actually do the zeroing. 188 // 189 //go:nowritebarrierrec 190 //go:nosplit 191 func wbZero(typ *_type, dst unsafe.Pointer) { 192 // This always copies a full value of type typ so it's safe 193 // to pass typ along as an optimization. See the comment on 194 // bulkBarrierPreWrite. 195 bulkBarrierPreWrite(uintptr(dst), 0, typ.PtrBytes, typ) 196 } 197 198 // wbMove performs the write barrier operations necessary before 199 // copying a region of memory from src to dst of type typ. 200 // Does not actually do the copying. 201 // 202 //go:nowritebarrierrec 203 //go:nosplit 204 func wbMove(typ *_type, dst, src unsafe.Pointer) { 205 // This always copies a full value of type typ so it's safe to 206 // pass a type here. 207 // 208 // See the comment on bulkBarrierPreWrite. 209 bulkBarrierPreWrite(uintptr(dst), uintptr(src), typ.PtrBytes, typ) 210 } 211 212 // reflect_typedmemmove is meant for package reflect, 213 // but widely used packages access it using linkname. 214 // Notable members of the hall of shame include: 215 // - gitee.com/quant1x/gox 216 // - github.com/goccy/json 217 // - github.com/modern-go/reflect2 218 // - github.com/ugorji/go/codec 219 // - github.com/v2pro/plz 220 // 221 // Do not remove or change the type signature. 222 // See go.dev/issue/67401. 223 // 224 //go:linkname reflect_typedmemmove reflect.typedmemmove 225 func reflect_typedmemmove(typ *_type, dst, src unsafe.Pointer) { 226 if raceenabled { 227 raceWriteObjectPC(typ, dst, getcallerpc(), abi.FuncPCABIInternal(reflect_typedmemmove)) 228 raceReadObjectPC(typ, src, getcallerpc(), abi.FuncPCABIInternal(reflect_typedmemmove)) 229 } 230 if msanenabled { 231 msanwrite(dst, typ.Size_) 232 msanread(src, typ.Size_) 233 } 234 if asanenabled { 235 asanwrite(dst, typ.Size_) 236 asanread(src, typ.Size_) 237 } 238 typedmemmove(typ, dst, src) 239 } 240 241 //go:linkname reflectlite_typedmemmove internal/reflectlite.typedmemmove 242 func reflectlite_typedmemmove(typ *_type, dst, src unsafe.Pointer) { 243 reflect_typedmemmove(typ, dst, src) 244 } 245 246 // reflectcallmove is invoked by reflectcall to copy the return values 247 // out of the stack and into the heap, invoking the necessary write 248 // barriers. dst, src, and size describe the return value area to 249 // copy. typ describes the entire frame (not just the return values). 250 // typ may be nil, which indicates write barriers are not needed. 251 // 252 // It must be nosplit and must only call nosplit functions because the 253 // stack map of reflectcall is wrong. 254 // 255 //go:nosplit 256 func reflectcallmove(typ *_type, dst, src unsafe.Pointer, size uintptr, regs *abi.RegArgs) { 257 if writeBarrier.enabled && typ != nil && typ.Pointers() && size >= goarch.PtrSize { 258 // Pass nil for the type. dst does not point to value of type typ, 259 // but rather points into one, so applying the optimization is not 260 // safe. See the comment on this function. 261 bulkBarrierPreWrite(uintptr(dst), uintptr(src), size, nil) 262 } 263 memmove(dst, src, size) 264 265 // Move pointers returned in registers to a place where the GC can see them. 266 for i := range regs.Ints { 267 if regs.ReturnIsPtr.Get(i) { 268 regs.Ptrs[i] = unsafe.Pointer(regs.Ints[i]) 269 } 270 } 271 } 272 273 // typedslicecopy should be an internal detail, 274 // but widely used packages access it using linkname. 275 // Notable members of the hall of shame include: 276 // - github.com/segmentio/encoding 277 // 278 // Do not remove or change the type signature. 279 // See go.dev/issue/67401. 280 // 281 //go:linkname typedslicecopy 282 //go:nosplit 283 func typedslicecopy(typ *_type, dstPtr unsafe.Pointer, dstLen int, srcPtr unsafe.Pointer, srcLen int) int { 284 n := dstLen 285 if n > srcLen { 286 n = srcLen 287 } 288 if n == 0 { 289 return 0 290 } 291 292 // The compiler emits calls to typedslicecopy before 293 // instrumentation runs, so unlike the other copying and 294 // assignment operations, it's not instrumented in the calling 295 // code and needs its own instrumentation. 296 if raceenabled { 297 callerpc := getcallerpc() 298 pc := abi.FuncPCABIInternal(slicecopy) 299 racewriterangepc(dstPtr, uintptr(n)*typ.Size_, callerpc, pc) 300 racereadrangepc(srcPtr, uintptr(n)*typ.Size_, callerpc, pc) 301 } 302 if msanenabled { 303 msanwrite(dstPtr, uintptr(n)*typ.Size_) 304 msanread(srcPtr, uintptr(n)*typ.Size_) 305 } 306 if asanenabled { 307 asanwrite(dstPtr, uintptr(n)*typ.Size_) 308 asanread(srcPtr, uintptr(n)*typ.Size_) 309 } 310 311 if goexperiment.CgoCheck2 { 312 cgoCheckSliceCopy(typ, dstPtr, srcPtr, n) 313 } 314 315 if dstPtr == srcPtr { 316 return n 317 } 318 319 // Note: No point in checking typ.PtrBytes here: 320 // compiler only emits calls to typedslicecopy for types with pointers, 321 // and growslice and reflect_typedslicecopy check for pointers 322 // before calling typedslicecopy. 323 size := uintptr(n) * typ.Size_ 324 if writeBarrier.enabled { 325 // This always copies one or more full values of type typ so 326 // it's safe to pass typ along as an optimization. See the comment on 327 // bulkBarrierPreWrite. 328 pwsize := size - typ.Size_ + typ.PtrBytes 329 bulkBarrierPreWrite(uintptr(dstPtr), uintptr(srcPtr), pwsize, typ) 330 } 331 // See typedmemmove for a discussion of the race between the 332 // barrier and memmove. 333 memmove(dstPtr, srcPtr, size) 334 return n 335 } 336 337 // reflect_typedslicecopy is meant for package reflect, 338 // but widely used packages access it using linkname. 339 // Notable members of the hall of shame include: 340 // - gitee.com/quant1x/gox 341 // - github.com/modern-go/reflect2 342 // - github.com/RomiChan/protobuf 343 // - github.com/segmentio/encoding 344 // - github.com/v2pro/plz 345 // 346 // Do not remove or change the type signature. 347 // See go.dev/issue/67401. 348 // 349 //go:linkname reflect_typedslicecopy reflect.typedslicecopy 350 func reflect_typedslicecopy(elemType *_type, dst, src slice) int { 351 if !elemType.Pointers() { 352 return slicecopy(dst.array, dst.len, src.array, src.len, elemType.Size_) 353 } 354 return typedslicecopy(elemType, dst.array, dst.len, src.array, src.len) 355 } 356 357 // typedmemclr clears the typed memory at ptr with type typ. The 358 // memory at ptr must already be initialized (and hence in type-safe 359 // state). If the memory is being initialized for the first time, see 360 // memclrNoHeapPointers. 361 // 362 // If the caller knows that typ has pointers, it can alternatively 363 // call memclrHasPointers. 364 // 365 // TODO: A "go:nosplitrec" annotation would be perfect for this. 366 // 367 //go:nosplit 368 func typedmemclr(typ *_type, ptr unsafe.Pointer) { 369 if writeBarrier.enabled && typ.Pointers() { 370 // This always clears a whole value of type typ, so it's 371 // safe to pass a type here and apply the optimization. 372 // See the comment on bulkBarrierPreWrite. 373 bulkBarrierPreWrite(uintptr(ptr), 0, typ.PtrBytes, typ) 374 } 375 memclrNoHeapPointers(ptr, typ.Size_) 376 } 377 378 // reflect_typedslicecopy is meant for package reflect, 379 // but widely used packages access it using linkname. 380 // Notable members of the hall of shame include: 381 // - github.com/ugorji/go/codec 382 // 383 // Do not remove or change the type signature. 384 // See go.dev/issue/67401. 385 // 386 //go:linkname reflect_typedmemclr reflect.typedmemclr 387 func reflect_typedmemclr(typ *_type, ptr unsafe.Pointer) { 388 typedmemclr(typ, ptr) 389 } 390 391 //go:linkname reflect_typedmemclrpartial reflect.typedmemclrpartial 392 func reflect_typedmemclrpartial(typ *_type, ptr unsafe.Pointer, off, size uintptr) { 393 if writeBarrier.enabled && typ.Pointers() { 394 // Pass nil for the type. ptr does not point to value of type typ, 395 // but rather points into one so it's not safe to apply the optimization. 396 // See the comment on this function in the reflect package and the 397 // comment on bulkBarrierPreWrite. 398 bulkBarrierPreWrite(uintptr(ptr), 0, size, nil) 399 } 400 memclrNoHeapPointers(ptr, size) 401 } 402 403 //go:linkname reflect_typedarrayclear reflect.typedarrayclear 404 func reflect_typedarrayclear(typ *_type, ptr unsafe.Pointer, len int) { 405 size := typ.Size_ * uintptr(len) 406 if writeBarrier.enabled && typ.Pointers() { 407 // This always clears whole elements of an array, so it's 408 // safe to pass a type here. See the comment on bulkBarrierPreWrite. 409 bulkBarrierPreWrite(uintptr(ptr), 0, size, typ) 410 } 411 memclrNoHeapPointers(ptr, size) 412 } 413 414 // memclrHasPointers clears n bytes of typed memory starting at ptr. 415 // The caller must ensure that the type of the object at ptr has 416 // pointers, usually by checking typ.PtrBytes. However, ptr 417 // does not have to point to the start of the allocation. 418 // 419 // memclrHasPointers should be an internal detail, 420 // but widely used packages access it using linkname. 421 // Notable members of the hall of shame include: 422 // - github.com/bytedance/sonic 423 // 424 // Do not remove or change the type signature. 425 // See go.dev/issue/67401. 426 // 427 //go:linkname memclrHasPointers 428 //go:nosplit 429 func memclrHasPointers(ptr unsafe.Pointer, n uintptr) { 430 // Pass nil for the type since we don't have one here anyway. 431 bulkBarrierPreWrite(uintptr(ptr), 0, n, nil) 432 memclrNoHeapPointers(ptr, n) 433 } 434