golang 内存一直涨排查实践

我们需要检查是否有 goroutine 泄漏。goroutine 泄漏会导致内存持续增长。检查 goroutine 的数量和 goroutine 的栈大小可以帮助我们找出问题所在。

Go 语言的内存管理机制是通过 goroutine 的栈来实现的。goroutine 的栈大小可以通过 `runtime.StackSize` 系统变量来设置。通常的设置是 2MB。

出现报警后，我们需要快速定位问题。溢出屏幕的 SRE 告警显示内存泄漏问题，请求量大了，业务场景也变多了，可能需要定时强制释放内存了。

我们可以通过 K8S 面板查看内存曲线，发现内存曲线长成了这样：每小时有一个峰值，因为我们有一个按小时执行的计划任务，可能会占用一些内存；之所以后来下降了好多，是因为我们重新部署了。

要确定是否是内存泄漏，我们需要通过 go pprof heap 查看内存使用情况。可以确定并不是内存泄漏：go pprof heap 前后对比区别不大，内存不是缓慢增长，而是突增一定水平，后台任务结束一段时间后，RSS 回到正常水平。

多出来的内存，在 go runtime heap 采样是看不到的！有可能是大量内存在申请、释放。而 cpu pprof 发现有一定量的 cpu 集中在 GC mem 上。

If this caught your attention, see: Golang Go

The first step in troubleshooting memory leaks in Go is to check for goroutine issues. This is because excessive goroutines are often the primary cause of memory leaks.

The initial investigation revealed that goroutines were not the problem, with a relatively low and stable number.

In fact, the goroutine count remained constant throughout the analysis, which ruled out excessive goroutines as the culprit.

Timer Usage Mismanagement is a common issue that can lead to memory leaks and other problems in Go programming.

Using time.After() incorrectly can cause memory leaks due to the creation of new timers that are not garbage collected until they expire.

In Go, time.Ticker needs to be manually stopped to prevent permanent memory leaks.

Explore further: Golang vs Go

Here are some tips to avoid timer usage mismanagement:

By following these best practices, you can write more efficient and reliable Go code.

When analyzing dump files, you can use the `go tool pprof` command to inspect the data. To do this, you can execute the command `go tool pprof http://localhost:6060/debug/pprof/profile?seconds=60` to collect data for 60 seconds. This will enter you into the command line interactive mode.

One of the most useful tools for analyzing dump files is the `go tool pprof` command. By executing this command, you can collect data for a specified amount of time and then inspect the results.

To analyze the data, you can use the `top` command to see the top functions consuming memory. This can give you a good idea of where the memory leak is coming from.

When analyzing dump files, it's essential to look for patterns and correlations between different functions and goroutines. By doing so, you can identify the root cause of the memory leak.

A unique perspective: Golang Command Line Arguments

To make the analysis process more efficient, you can use the `heap` command to see the memory allocation patterns. This can help you identify the objects that are causing the memory leak.

Here are some common commands you can use to analyze dump files:

By using these commands, you can gather valuable insights into the memory leak and identify the root cause.

排查过程中，我们发现Go程序的内存泄漏主要是由于goroutine泄漏导致的。Go runtime会自动管理goroutine，但是如果goroutine被阻塞，runtime也会被阻塞，导致内存泄漏。

使用Go的pprof工具，我们可以分析程序的内存泄漏情况。通过pprof，我们发现程序中存在大量的goroutine泄漏，导致内存一直在涨。

使用Go的runtime.SetGCPercent函数，我们可以调节Go的gc频率，从而减少内存泄漏。通过设置gc频率，我们可以减少goroutine泄漏的发生率。

在排查过程中，我们还发现程序中存在大量的不必要的内存分配和回收。通过优化程序的内存分配和回收，例如使用sync.Pool和sync.Once，我们可以显著减少内存泄漏。

If this caught your attention, see: Sync Golang

一种方法是通过环境变量指定 GODEBUG=madvdontneed=1，这样可以强制 runtime 继续使用 MADV_DONTNEED。但是，这个方法需要谨慎使用，因为它会触发 TLB shootdown，以及更多的 page fault，可能会对延迟敏感的业务造成影响。

升级 go 编译器版本到 1.16 以上可以解决问题，因为 go 1.16 已经放弃了使用 MADV_DONTNEED 的 GC 策略，改为了及时释放内存而不是等到内存有压力时的惰性释放。这是一个更为合适的选择，多数情况下都是如此。

手动调用 debug.FreeOSMemory 可以解决 pprof 看 heap 使用的内存小于 RSS 很多的问题，但是执行这个操作是有代价的。

一种解决方法是通过环境变量指定 GODEBUG=madvdontneed=1 来强制 runtime 继续使用 MADV_DONTNEED。这种方法可以参考 GitHub 上的 Issue #28466。

升级 Go 编译器版本到 1.16 以上是一个更好的选择，因为 Go 1.16 已经放弃了 MADV_DONTNEED 策略，改为了及时释放内存。

升级 Go 编译器版本到 1.16 以上是因为 Go 官网认为及时释放内存的方式更加可取。

升级 Go 编译器版本之前，请注意 Go1.13 版本中的 FreeOSMemory 不起作用（参考 GitHub 上的 Issue #35858）。

判断 goroutine 问题是解决问题的关键步骤之一。

Expand your knowledge: How to Update a Github Using Golang

We chose solution two and saw a significant improvement after upgrading to go1.16, with no more rapid memory growth issues.

The memory usage of the instance stabilized, and we were able to identify changes in the functions consuming memory using pprof.

The metrics.glob function, which was previously a major memory hog, saw a significant decrease in memory usage.

goroutine申请过多会导致内存泄露，因为增长速度快于释放速度，goroutine越来越多。例如，业务请求量大时，新建一个client来处理请求，来不及释放。

I/O问题也会导致goroutine阻塞，例如网络连接未设置超时时间，导致goroutine一直在等待。

互斥锁未释放也会导致goroutine阻塞，例如goroutineA对共享变量加锁但未释放，导致其他goroutine无法获取到锁资源。

使用全局map也可能导致内存泄露，因为map在删除Key的时候，只会把Key标记为empty，而不是彻底删除。

使用定时器不当也会导致内存泄漏，例如time.After()使用不当，会产生NewTimer()，在duration x到期之前，新创建的timer不会被GC。

使用slice引起内存泄露，例如两个slice共享地址，其中一个为全局变量，另一个也无法被gc。

获取长字符串中的一段导致长字符串未释放，例如变量s0是包级别的变量，而函数f()接受一个参数s1，在函数f()中，将s1的前50个字符赋值给s0。

使用time.Ticker未stop也会导致内存泄漏，建议总是建议在for之外初始化一个定时器，并且for结束时手工stop一下定时器。

以下是导致内存泄露的常见原因：

To tackle the issue of golang 内存一直涨, it's essential to have a solid approach. First, observe your server instance and check the memory usage to confirm the problem.

You can do this by clicking on the instance list on the tce platform or checking the runtime monitoring on the ms platform. Both methods will give you an overview of your memory usage.

To further investigate, you can use pprof to sample and judge the goroutine count. If it's abnormal, use pprof's graph and source features to locate the specific code lines causing the issue. Be sure to check for problems like select blocking, channel blocking, or slice misuse in your code logic, and also consider if the framework has any unreasonable places.

Discover more: Golang Code Comment Specifications

When you encounter a Go memory leak problem, you can follow a specific approach to troubleshoot and resolve it.

First, observe your server instance to see if there's a memory leak issue. You can do this by checking the memory usage on the TCE platform or the MS platform's runtime monitoring.

Broaden your view: Google Cloud Platform Golang

To check the memory usage, click on the "Instance List" on the TCE platform, or go to the "Runtime Monitoring" on the MS platform.

You can also use the monitoring tools mentioned earlier to observe the goroutine count and use pprof to take samples and determine if the goroutine count has increased abnormally.

To further investigate, use pprof to identify the specific code lines causing the issue, and then examine the entire call chain to see if there are any problems like select blocking, channel blocking, or slice misuse.

Here's a step-by-step guide to help you troubleshoot memory leaks:

Once you've fixed the issue, deploy it to production and monitor the memory usage again to ensure the problem is resolved.

Scavenging is a crucial process in Go that helps manage memory usage, but it can be tricky to observe its behavior, especially when dealing with large data structures. Scavenging occurs both in the background and synchronously when the heap grows.

Scavenging can either return too little or too much memory, both of which can cause issues. Returning too little memory can lead to OOM (Out of Memory) risks, while returning too much memory can cause performance degradation due to frequent heap lock acquisitions.

To mitigate these issues, Go has implemented smarter scavenging and a soft memory limit. Smarter scavenging aims to strike a balance between returning too little and too much memory, while the soft memory limit helps prevent memory exhaustion.

Here's a summary of the scavenging process:

In the case of heap growth scavenging, Go calculates the excess memory value based on the heap's alloc growth, growth, and RSS occupation. However, due to the high cost of acquiring the heap lock, Go uses a heuristic to determine the scavenging size, bytesToScavenge. If bytesToScavenge is 0, scavenging is not performed.

In Go, a common scenario that can lead to memory leaks is using WaitGroup incorrectly. WaitGroup is a goroutine manager that needs to know how many goroutines are working for it and when they're done. If the Add, Done, and wait numbers don't match, WaitGroup will wait indefinitely.

WaitGroup will wait until it receives enough Done() signals, but if it's not notified correctly, it can cause problems. For example, if WaitGroup is initialized with Add(2) but only one goroutine is done, WaitGroup will wait forever. This is because it's waiting for two Done() signals, but it only received one.

GOGC 和 go mem limit 可以通过调节来控制 Go 程序对 GC 的积极性和粒度。

Go Memory Ballast 技巧可以通过强行创建一个大数组，令 Go GC 迟钝，从而避免频繁 GC 引起性能抖动。

GOGC 和 go mem limit 可以配合使用，参考 go memory tuning 的官方文档可以了解更多。

开启 soft mem limit 时，goal 的计算公式是：

\[

\text{{goal}} = \frac{{100 - \text{{reduceExtraPercent}}}}{{100}} \times \text{{memoryLimit}}

\]

其中，reduceExtraPercent 是一个稍小的常数，取 5。让实际参与计算的 memoryLimit 更小，从而在逼近内存极限时更积极地工作。

在开启 GOGC 时，runtime/mgcpacer.go 是相关的源码文件。