Debugging concurrency programs in go
1 like357 views
The document discusses the challenges of debugging concurrent programs in Go, highlighting techniques for effectively managing and diagnosing issues with goroutines and channels. It includes practical examples of setting breakpoints, using debugging tools like Delve and GDB, and specific strategies to avoid deadlocks. Additionally, it provides references for further reading and guidelines for debugging concurrency applications.
![@a_soldatenko
How to debug channel?(dlv) p ch
chan int {
qcount: 0,
dataqsiz: 4,
buf: *[4]int [0,0,0,0],
elemsize: 8,
closed: 0,
elemtype: *runtime._type {size: 8, ptrdata: 0, hash: 4149441018, tflag: tflagUncommon|tflagExtraStar|
tflagNamed|tflagRegularMemory (15), align: 8, fieldAlign: 8, kind: 2, equal: runtime.memequal64, gcdata: *1,
str: 663, ptrToThis: 22432},
sendx: 0,
recvx: 0,
recvq: waitq<int> {
first: *sudog<int> nil,
last: *sudog<int> nil,},
sendq: waitq<int> {
first: *sudog<int> nil,
last: *sudog<int> nil,},
lock: runtime.mutex {key: 0},}
(dlv) n
> main.main() ./main_ch.go:7 (PC: 0x105e8d9)
2:
3:
4: func main() {
5: ch := make(chan int, 4)
6: ch <- 1
=> 7: ch <- 2
8: ch <- 3
9: ch <- 4
10: close(ch)
11: }
(dlv) p ch](https://image.slidesharecdn.com/debuggingconcurrencyprogramsingo-201208003003/85/Debugging-concurrency-programs-in-go-24-320.jpg)
![@a_soldatenko
How to detect deadlocks
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/Users/andrii/workspace/src/github.com/andriisoldatenko/
gowayfest/main_deadlock.go:5 +0x50
exit status 2](https://image.slidesharecdn.com/debuggingconcurrencyprogramsingo-201208003003/85/Debugging-concurrency-programs-in-go-31-320.jpg)
Debugging concurrency programs in go
- 2. Andrii Soldatenko • Gopher, OSS contributor • father of 👧 👶 • debuggers fan • Speaker at many conferences @a_soldatenko
- 5. @a_soldatenko Debugging sequential programs dlv test -- -test.run TestFibonacciBig (dlv) b main_test.go:6 Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/ debug_test.TestFibonacciBig() ./main_test.go:6 (dlv) c > github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./ main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f) 1: package main 2: 3: import "testing" 4: 5: func TestFibonacciBig(t *testing.T) { => 6: var want int64 = 55 7: got := FibonacciBig(10) 8: if got.Int64() != want { 9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d, want: %d", 10, got.Int64(), want) 10: } 11: } (dlv)
- 6. @a_soldatenko Debugging concurrent programs package main import ( "fmt" "time" ) func say(s string) { for i := 0; i < 5; i++ { time.Sleep(100 * time.Millisecond) fmt.Println(s) } } func main() { go say("world") say("hello") } gowayfest git:(master) ✗ go run main.go hello world world hello hello world world hello world hello gowayfest git:(master) ✗ go run main.go world hello hello world hello world world hello hello world
- 8. What is a goroutine? https://tpaschalis.github.io/goroutines-size/
- 9. Do you know why gorotines?
- 10. How can I debug concurrent Go program?
- 12. visualize goroutines🐞 ? func main() { c := coloredgoroutine.Colors(os.Stdout) fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from main routine") count := 10 var wg sync.WaitGroup wg.Add(count) for i := 0; i < count; i++ { i := i go func() { fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from loop i =", i) wg.Done() }() } wg.Wait() }
- 15. Print scheduling events? $ GODEBUG=schedtrace=5000 <binary>
- 17. @a_soldatenko - delve - gdb using debuggers:
- 18. Advanced debugging techniques of Go code
- 19. @a_soldatenko How to set breakpoint inside goroutine?package main import ( "fmt" ) func say(s string, r chan string) { fmt.Println(s) r <- s } func main() { chan1 := make(chan string) chan2 := make(chan string) go say("world", chan1) go say("hello", chan2) res1 := <-chan1 res2 := <-chan2 fmt.Printf("Channel 1: %snChannel 2: %sn", res1, res2) }
- 20. @a_soldatenko > main.say() ./main.go:9 (hits goroutine(7):1 total:1) (PC: 0x10c46c9) 4: "fmt" 5: ) 6: 7: func say(s string, r chan string) { 8: fmt.Println(s) => 9: r <- s 10: } 11: 12: func main() { 13: chan1 := make(chan string) 14: chan2 := make(chan string) How to set breakpoint inside goroutine?
- 21. @a_soldatenko
- 22. @a_soldatenko How to debug channel? package main func main() { ch := make(chan int, 4) ch <- 1 ch <- 2 ch <- 3 ch <- 4 close(ch) }
- 23. @a_soldatenko
- 24. @a_soldatenko How to debug channel?(dlv) p ch chan int { qcount: 0, dataqsiz: 4, buf: *[4]int [0,0,0,0], elemsize: 8, closed: 0, elemtype: *runtime._type {size: 8, ptrdata: 0, hash: 4149441018, tflag: tflagUncommon|tflagExtraStar| tflagNamed|tflagRegularMemory (15), align: 8, fieldAlign: 8, kind: 2, equal: runtime.memequal64, gcdata: *1, str: 663, ptrToThis: 22432}, sendx: 0, recvx: 0, recvq: waitq<int> { first: *sudog<int> nil, last: *sudog<int> nil,}, sendq: waitq<int> { first: *sudog<int> nil, last: *sudog<int> nil,}, lock: runtime.mutex {key: 0},} (dlv) n > main.main() ./main_ch.go:7 (PC: 0x105e8d9) 2: 3: 4: func main() { 5: ch := make(chan int, 4) 6: ch <- 1 => 7: ch <- 2 8: ch <- 3 9: ch <- 4 10: close(ch) 11: } (dlv) p ch
- 25. @a_soldatenko dlv send to channel value similar to set variable.
- 27. @a_soldatenko Gdb and golang go build -ldflags=-compressdwarf=false - gcflags=all="-N -l" -o main main.go
- 30. @a_soldatenko Deadlocks happen and are painful to debug.
- 31. @a_soldatenko How to detect deadlocks fatal error: all goroutines are asleep - deadlock! goroutine 1 [chan send]: main.main() /Users/andrii/workspace/src/github.com/andriisoldatenko/ gowayfest/main_deadlock.go:5 +0x50 exit status 2
- 32. @a_soldatenko Real world examples complicated scenario & tools https://github.com/sasha-s/go-deadlock https://github.com/cockroachdb/cockroach/issues/7972
- 33. @a_soldatenko
- 34. @a_soldatenko 7 simple rules for debugging concurrency applications - Never assume a particular order of execution - Implement concurrency at the highest level possible - Don’t forget Go only detects when the program as a whole freezes, not when a subset of goroutines get stuck. - STRACE
- 35. @a_soldatenko 7 simple rules for debugging concurrency applications - conditional breakpoints your best friend - DEBUG=schedtrace=5000 - go-deadlock
- 36. @a_soldatenko References 1 - https://github.com/golang/go/blob/release- branch.go1.14/src/runtime/HACKING.md - https://github.com/golang/go/wiki/LearnConcurrency - https://rakyll.org/go-cloud/ - https://yourbasic.org/golang/detect-deadlock/
- 37. @a_soldatenko References 2 - https://blog.minio.io/debugging-go-routine-leaks- a1220142d32c - https://golang.org/src/cmd/link/internal/ld/dwarf.go - https://golang.org/src/runtime/runtime-gdb.py - https://cseweb.ucsd.edu/~yiying/GoStudy- ASPLOS19.pdf - https://golang.org/doc/articles/race_detector.html