runtime: segfault during conservative scan of asynchronously preempted goroutine

[jamesl33]

What version of Go are you using (go version)?

$ go version
go version go1.14.1 linux/amd64

Does this issue reproduce with the latest release?

Yes (but not consistently) - We have reproductions up to 1.14.3 (and have just
updated to 1.14.4 but no tests have been run as of yet).

What operating system and processor architecture are you using?

CentOS 7 amd64 - E5-2630 v2 (24 vCPU)

What issue are we seeing?

From a brief look at the stacktrace and runtime it looks like we are currently
seeing a segfault during the conservative scan of an asynchronously preempted
goroutine. While we have only seen this issue since updating to 1.14.1 (we
skipped 1.14) we do rely on a couple of libraries that make use of 'unsafe' so
we wouldn't be surprised if this was due to the misuse of 'unsafe' rather than
an issue with the runtime itself.

I've included the full stacktrace below along with a snippet from the same
backtrace displaying what I've described above. Any help debugging this issue
would be greatly appreciated whether that be tips on which GODEBUG settings to
use so that we can get more information about why this is happening or some
steps we could take to debug this issue/to provide you with extra information.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f2f9047b8ef pc=0x42f616]

runtime stack:
runtime.throw(0xbfd246, 0x2a)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/panic.go:1114 +0x72
runtime.sigpanic()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/signal_unix.go:679 +0x46a
runtime.(*mspan).isFree(...)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mbitmap.go:255
runtime.scanConservative(0xc002b9fbd8, 0x178, 0x0, 0xc00004a698, 0x7f2f65cb3348)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:1368 +0xf6
runtime.scanframeworker(0x7f2f65cb3238, 0x7f2f65cb3348, 0xc00004a698)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:875 +0x29d
runtime.scanstack.func1(0x7f2f65cb3238, 0x0, 0x13c7920)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:736 +0x3d
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc0006f1500, 0x0, 0x0, 0x7fffffff, 0x7f2f65cb3330, 0x0, 0x0, ...)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/traceback.go:334 +0x110e
runtime.scanstack(0xc0006f1500, 0xc00004a698)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:739 +0x15e
runtime.markroot.func1()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:226 +0xbf
runtime.markroot(0xc00004a698, 0x153)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:199 +0x2f3
runtime.gcDrainN(0xc00004a698, 0x10000, 0x10000)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:1119 +0xff
runtime.gcAssistAlloc1(0xc000c84d80, 0x10000)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:531 +0xf3
runtime.gcAssistAlloc.func1()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/mgcmark.go:442 +0x33
runtime.systemstack(0x0)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.1/go/src/runtime/proc.go:1041

stack_trace.txt

[ALTree]

cc @aclements @mknyszek

[randall77]

This does look like corruption of internal runtime data structures.

[aclements]

You mentioned possible misuse of unsafe. Could you compile with -gcflags=all=-d=checkptr (note that this is implied if you're already using -race or -msan)?

Can you reproduce this with GOTRACEBACK=system or GOTRACEBACK=crash (the latter is even more verbose, but will send around lots of signals when it crashes, which can in principle make have bad effects)? These will include SP values in the traceback, which should let us match up exactly which call frame is being scanned and where it's stopped.

[jamesl33]

Thank you for the information. I've got a build running with the requested gcflags and have got the GOTRACEBACK environment variable set to crash. I'll update the issue once I've been able to reproduce the segfault.

[jamesl33]

We've been able to reproduce this issue on 1.14.4 with the gcflags set as requested, however, our Jenkins environment didn't have the GOTRACEBACK variable set as we'd intended (we run multiple Jenkins instances and the reproduction is not running on our own). I've re-setup the environment on the performance machine and will continue running until I can provide an extended stack trace and hopefully a core dump.

For the time being I've attached the complete stack traces for reproductions that we have so far. stack_trace-1.txt is noteworthy because we appear fail in a different location in the runtime.

stack_trace-1.txt, stack_trace-2.txt, stack_trace-3.txt

[jamesl33]

I've got a reproduction of the segfault with the requested gcflags and the GOTRACEBACK environment variable set to include runtime created goroutines, however, it does appear that, as in stack_trace-1.txt we have failed in a different location than in the other stack traces. If there's anything else you require to debug the issue please let me know and I'll do my best to provide it.

I'll leave the existing environment setup and update the issue whenever we come across any more reproductions. It's also worth noting that we have since downgraded a branch to Go 1.13.12 (and continued running testing) and we haven't yet encountered this issue.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f940c242512 pc=0x42fa4a]

runtime stack:
runtime.throw(0xc26236, 0x2a)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/panic.go:1116 +0x72 fp=0x7f93d37fbe68 sp=0x7f93d37fbe38 pc=0x443502
runtime.sigpanic()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/signal_unix.go:679 +0x46a fp=0x7f93d37fbe98 sp=0x7f93d37fbe68 pc=0x459cfa
runtime.markBits.setMarked(...)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mbitmap.go:295
runtime.greyobject(0xc0011a5100, 0xc0031efcd8, 0x1baf0, 0x7f93e00b5db8, 0xc000031698, 0x99b10)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:1439 +0x22a fp=0x7f93d37fbec8 sp=0x7f93d37fbe98 pc=0x42fa4a
runtime.scanConservative(0xc0031efcd8, 0x20068, 0x0, 0xc000031698, 0x7f93d37fc350)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:1374 +0x15d fp=0x7f93d37fbf18 sp=0x7f93d37fbec8 pc=0x42f69d
runtime.scanframeworker(0x7f93d37fc240, 0x7f93d37fc350, 0xc000031698)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:875 +0x29d fp=0x7f93d37fbfa8 sp=0x7f93d37fbf18 pc=0x42e99d
runtime.scanstack.func1(0x7f93d37fc240, 0x0, 0x1413ec0)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:736 +0x3d fp=0x7f93d37fbfd0 sp=0x7f93d37fbfa8 pc=0x46eebd
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc00011ad80, 0x0, 0x0, 0x7fffffff, 0x7f93d37fc338, 0x0, 0x0, ...)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/traceback.go:334 +0x110e fp=0x7f93d37fc2a8 sp=0x7f93d37fbfd0 pc=0x467d7e
runtime.scanstack(0xc00011ad80, 0xc000031698)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:739 +0x15e fp=0x7f93d37fc4b0 sp=0x7f93d37fc2a8 pc=0x42e07e
runtime.markroot.func1()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:226 +0xbf fp=0x7f93d37fc500 sp=0x7f93d37fc4b0 pc=0x46ed5f
runtime.markroot(0xc000031698, 0xc3)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:199 +0x2f3 fp=0x7f93d37fc580 sp=0x7f93d37fc500 pc=0x42d113
runtime.gcDrain(0xc000031698, 0x3)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgcmark.go:999 +0x107 fp=0x7f93d37fc5d8 sp=0x7f93d37fc580 pc=0x42ead7
runtime.gcBgMarkWorker.func2()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/mgc.go:1940 +0x80 fp=0x7f93d37fc618 sp=0x7f93d37fc5d8 pc=0x46eb70
runtime.systemstack(0x0)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/asm_amd64.s:370 +0x66 fp=0x7f93d37fc620 sp=0x7f93d37fc618 pc=0x471816
runtime.mstart()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.14.4/go/src/runtime/proc.go:1041 fp=0x7f93d37fc628 sp=0x7f93d37fc620 pc=0x4482e0

stack_trace-4.txt

[jamesl33]

Just giving an update, we've just updated to Go 1.15 and will continue running our performance testing to see if we can reproduce the issue further (the performance cluster we were reproducing on recently had some issues/downtime). With that said, is there anything else we can provide to help debug the issue?

[YuhangMa1117]

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f024c1505ab pc=0x4226d4]

runtime stack:
runtime.throw({0x11d71fc?, 0x1c97f9670?})
	/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f01c97f9640 sp=0x7f01c97f9610 pc=0x439b5d
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7f01c97f9690 sp=0x7f01c97f9640 pc=0x450409
runtime.(*mspan).isFree(...)
	/usr/local/go/src/runtime/mbitmap.go:231
runtime.scanConservative(0xc00055c960, 0x1088, 0x0, 0x46dea1?, 0x7f01c97f9b68)
	/usr/local/go/src/runtime/mgcmark.go:1432 +0x134 fp=0x7f01c97f96e8 sp=0x7f01c97f9690 pc=0x4226d4
runtime.scanframeworker(0x7f01c97f9a80, 0x7f01c97f9b68, 0x1?)
	/usr/local/go/src/runtime/mgcmark.go:941 +0x148 fp=0x7f01c97f9750 sp=0x7f01c97f96e8 pc=0x421968
runtime.scanstack.func1(0x1ad08e0?, 0x1b47760?)
	/usr/local/go/src/runtime/mgcmark.go:801 +0x25 fp=0x7f01c97f9778 sp=0x7f01c97f9750 pc=0x4217e5
runtime.gentraceback(0x20?, 0x1366128?, 0x0?, 0x7f01c97f9b68?, 0x0, 0x0, 0x7fffffff, 0x7f01c97f9cc8, 0x0?, 0x0)
	/usr/local/go/src/runtime/traceback.go:334 +0xd0d fp=0x7f01c97f9ae8 sp=0x7f01c97f9778 pc=0x4602ed
runtime.scanstack(0xc0026489c0, 0xc000055740)
	/usr/local/go/src/runtime/mgcmark.go:804 +0x1da fp=0x7f01c97f9cf0 sp=0x7f01c97f9ae8 pc=0x42125a
runtime.markroot.func1()
	/usr/local/go/src/runtime/mgcmark.go:240 +0xc5 fp=0x7f01c97f9d40 sp=0x7f01c97f9cf0 pc=0x420085
runtime.markroot(0xc000055740, 0x11b6, 0x1)
	/usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7f01c97f9de0 sp=0x7f01c97f9d40 pc=0x41fd25
runtime.gcDrain(0xc000055740, 0x2)
	/usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7f01c97f9e40 sp=0x7f01c97f9de0 pc=0x421dff
runtime.gcBgMarkWorker.func2()
	/usr/local/go/src/runtime/mgc.go:1323 +0x154 fp=0x7f01c97f9e90 sp=0x7f01c97f9e40 pc=0x41e3d4
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:492 +0x49 fp=0x7f01c97f9e98 sp=0x7f01c97f9e90 pc=0x46bce9

@mknyszek we caught a panic whose stack trace similar to the above, the stack may be helpful. The go version is 1.19