Go tooling

• go tool pprof -web ~/x.profile
• go tool trace
  The package also exports a handler that serves execution trace data for the "go tool trace" command. To collect a 5-second execution trace:

wget -O trace.out http://localhost:6060/debug/pprof/trace?seconds=5
go tool trace trace.out

• 内存泄漏
  • 导出时间点1堆的profile：curl -s http://127.0.0.1:8080/debug/pprof/heap > base.heap
  • 导出时间点2堆的profile：curl -s http://127.0.0.1:8080/debug/pprof/heap > current.heap
  • go tool pprof --base base.heap current.heap

When setting nodefraction=0 we will get to see the entire map of the allocated objects, including the smaller ones.

list can find the source code when searching for it under your GOPATH environment. In cases where the root it is searching for does not match, which depends on your build machine, you can use the -trim_path option. This will assist with fixing it and letting you see the annotated source code. Remember to set your git to the right commit which was running when the heap profile was captured.

如果想要了解对应的汇编代码，可以使用 disadm <regex> 命令。这两个命令虽然强大，但是在命令行中查看代码并不是很方面，所以可以使用 weblist <regex> 命令，用法和两者一样，但它会在浏览器打开一个页面，能够同时显示源代码和汇编代码。

• 可以执行o命令查询当前options信息。

(pprof) o
  call_tree                 = false
  compact_labels            = true
  cumulative                = flat                 //: [cum | flat]
  divide_by                 = 1
  drop_negative             = false
  edgefraction              = 0.001
  focus                     = ""
  granularity               = filefunctions        //: [addresses | filefunctions | files | functions | lines]
  hide                      = ""
  ignore                    = ""
  mean                      = false
  nodecount                 = -1                   //: default
  nodefraction              = 0.005
  noinlines                 = false
  normalize                 = false
  output                    = ""
  prune_from                = ""
  relative_percentages      = false
  sample_index              = delay                //: [contentions | delay]
  show                      = ""
  show_from                 = ""
  tagfocus                  = ""
  taghide                   = ""
  tagignore                 = ""
  tagshow                   = ""
  trim                      = true
  trim_path                 = ""
  unit                      = minimum

• 在执行go tool pprof 时可以指定host:port访问heap信息。
  pprof支持交互的web 接口: Added an interactive web interface (#154)
  

go tool pprof -http=':' http://10.252.3.53:8051/debug/pprof/heap

• Mutex和Block profile默认是disable的。

• block: Block profile shows where goroutines block waiting on synchronization primitives (including timer channels). Block profile is not enabled by default; use runtime.SetBlockProfileRate to enable it.
• mutex: Mutex profile reports the lock contentions. When you think your CPU is not fully utilized due to a mutex contention, use this profile. Mutex profile is not enabled by default, see runtime.SetMutexProfileFraction to enable it.

(ENV_3.7.6) ➜ /home/admin/go/src ☞ git:(bug/update_mig) ✗  go tool pprof http://localhost:6060/debug/pprof/block
Fetching profile over HTTP from http://localhost:6060/debug/pprof/block
Saved profile in /home/admin/pprof/pprof.test_wait_group2.contentions.delay.007.pb.gz
File: test_wait_group2
Type: delay
Time: Dec 27, 2020 at 9:59am (EST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 4s, 100% of 4s total
      flat  flat%   sum%        cum   cum%
        2s 50.00% 50.00%         2s 50.00%  sync.(*WaitGroup).Wait
        2s 50.00%   100%         2s 50.00%  runtime.chanrecv1
         0     0%   100%         2s 50.00%  main.main
         0     0%   100%         2s 50.00%  main.testWaitGroup
         0     0%   100%         2s 50.00%  runtime.main
(pprof) list main.main
Total: 4s
ROUTINE ======================== main.main in /home/admin/go/src/github.com/microyahoo/go-exercises/test_wait_group2.go
         0         2s (flat, cum) 50.00% of Total
         .          .     27:
         .          .     28:   go testWaitGroup()
         .          .     29:
         .          .     30:   for {
         .          .     31:           select {
         .         2s     32:           case <-ticker.C:
         .          .     33:                   fmt.Println("hello")
         .          .     34:           }
         .          .     35:   }
         .          .     36:
         .          .     37:   // for range ticker.C {
(pprof) traces
File: test_wait_group2
Type: delay
Time: Dec 27, 2020 at 9:59am (EST)
-----------+-------------------------------------------------------
        2s   sync.(*WaitGroup).Wait
             main.testWaitGroup
-----------+-------------------------------------------------------
        2s   runtime.chanrecv1
             main.main
             runtime.main
-----------+-------------------------------------------------------
(pprof)

GC

GODEBUG=gctrace=1 prints garbage collector events at each collection, summarizing the amount of memory collected and the length of the pause.
GODEBUG=schedtrace=X prints scheduling events every X milliseconds.

(ENV_3.7.6) 🍺 /home/admin/go-exercises ☞ git:(master) ✗ GODEBUG=gctrace=1 ./gc1
2020/12/27 09:57:16 215584
2020/12/27 09:57:16 215584
2020/12/27 09:57:16 215584
2020/12/27 09:57:16 66781184
gc 1 @0.003s 17%: 1.4+3.2+0.066 ms clock, 11+0.094/0.11/0+0.53 ms cpu, 95->95->95 MB, 96 MB goal, 8 P
gc 2 @0.009s 22%: 0.71+0.35+0.045 ms clock, 5.6+0.085/0.22/0.011+0.36 ms cpu, 191->191->95 MB, 192 MB goal, 8 P
gc 3 @0.010s 66%: 14+0.16+0.005 ms clock, 116+0.048/0.20/0+0.047 ms cpu, 191->191->95 MB, 192 MB goal, 8 P
gc 4 @0.027s 73%: 13+0.32+0.006 ms clock, 111+0.18/0.20/0.099+0.048 ms cpu, 190->190->95 MB, 191 MB goal, 8 P
gc 5 @0.041s 73%: 6.4+2.0+0.004 ms clock, 51+0.041/0.17/0+0.036 ms cpu, 190->190->95 MB, 191 MB goal, 8 P
gc 6 @0.050s 76%: 6.0+0.23+0.005 ms clock, 48+0.14/0.14/0.037+0.041 ms cpu, 190->190->95 MB, 191 MB goal, 8 P
gc 7 @0.057s 76%: 5.6+1.0+0.005 ms clock, 45+0.11/0.13/0.030+0.044 ms cpu, 191->286->190 MB, 192 MB goal, 8 P
gc 8 @0.071s 68%: 0.006+0.23+0.004 ms clock, 0.050+0/0.19/0.13+0.037 ms cpu, 381->381->0 MB, 382 MB goal, 8 P
2020/12/27 09:57:16 400233568
2020/12/27 09:57:16 1000457904
2020/12/27 09:57:16 400233568
2020/12/27 09:57:16 469303296

目前格式是：Golang Environment Variables

gc # @#s #%: #+#+# ms clock, #+#/#/#+# ms cpu, #->#-># MB, # MB goal, # P

其中字段含义如下（'#' 代表数字）：

  gc #             每次 GC 时递增的 GC 编号
  @#s              程序运行的秒数
  #%               自程序启动后在 GC 中花费的时间百分比
  #+...+#          GC 各阶段的挂钟时间（wall-clock）/CPU 时间
  #->#-># MB       GC 开始时的堆大小、GC 结束时的堆大小和活动堆大小
  # MB goal        目标堆大小
  # P              使用的处理器数量

这些阶段是 stop-the-world（STW）清除终止（sweep termination），并发标记和扫描，以及 STW 标记终止（mark termination）。
mark/scan 的 CPU 时间被分解为辅助时间（根据分配执行的 GC）、后台 GC 时间和空闲 GC 时间。
如果该行以 "(forced)" 结尾，则此 GC 由 runtime.GC() 调用强制执行。

Debug

we recommend disabling optimizations when building the code being debugged. The following command builds a package with no compiler optimizations:

$ go build -gcflags=all="-N -l"

As part of the improvement effort, Go 1.10 introduced a new compiler flag -dwarflocationlists. The flag causes the compiler to add location lists that helps debuggers work with optimized binaries. The following command builds a package with optimizations but with the DWARF location lists:

$ go build -gcflags="-dwarflocationlists=true"

gRPC

Turn on logging for the client and server
For go client:

GRPC_GO_LOG_VERBOSITY_LEVEL=99 
GRPC_GO_LOG_SEVERITY_LEVEL=info

For C++ server:

GRPC_VERBOSITY=DEBUG 
GRPC_TRACE=server_channel

References

• golang pprof
• golang diagnostics
• Golang Environment Variables
• Data Race Detector
• Go环境变量
• Golang core dump debugging
• go 工具使用 tips
• Go 程序调试
• Linux Perf
• Debugging Containers With Stripped Binaries In Go
• golang gdb
• delve
• The tips of using gdb to debug Golang program
• 使用gdb远程调试Go代码（arm）
• Security restriction for auto-loading
• go tool pprof
• How I investigated memory leaks in Go using pprof on a large codebase
• 使用 pprof 和火焰图调试 golang 应用
• BPF和Go：在Linux中内省的现代方式[译]
• go pprof 性能分析
• Debugging Containers With Stripped Binaries In Go
• Unavailable error when MaxConnectionAge and MaxConnectionAgeGrace is enabled on the server