We've built a beat against the latest libbeat and we found that over time the RSS memory and heap allocations, as reported by go, steadily increased and wasn't freed.
Doing some analysis I tracked it down to the underlying bytes.Buffer used in the elasticsearch output encoders.
When the buffer is reset it retains the allocated memory to reduce the costs of allocating it again during a future iteration. This seems like a reasonable approach if you have consistent buffer sizes, however we have a workload that generates a sparse distribution of log sizes with a few infrequent but extremely large logs. This causes bytes.Buffer to allocate a large chunk of memory and never free it. This is also exacerbated by the fact that the underlying byte slice is grown by doubling it each time, and that this process repeats itself multiple times, at least once per output goroutine.
All of the above together leads to multiple overly large buffers allocated on the heap and never being freed.
I hacked in the following implementation of the reset function which creates a new buffer if the capacity of the existing buffer exceeds 10mb, as the majority of our logs easily fit within 10mb:
func (b *jsonEncoder) Reset() {
if b.buf.Cap() > 10*1024*1024 {
b.buf = bytes.NewBuffer(make([]byte, 10*1024*1024))
b.resetState()
return
}
b.buf.Reset()
}
This significantly reduce the memory usage of our beat. Where our beat was previously allocating 5GB of memory, which never dropped back. With the above change we're now seeing peak allocation of under 1.5GB but, more importantly, an average heap of less than 200mb.
It's a fairly simple change which I think many of the consumers of libbeat would benefit from as it's currently implemented. The gzip encoder may also benefit from this change?
A more sophisticated implementation would be to track the distribution of buffer sizes at runtime to settle on an optimal maximum buffer size. I did start drafting an implementation for this, but I'm not sure it's worth the added complexity and overhead for the bookkeeping. A halfway house would be to add it as a configurable value to allow people to tune it for their needs. I could see this being useful for people with a workload that typically had events larger than the static value chosen at compile time.
I was going to raise this on the github issue tracker as a bug, however it linked me here to get it confirmed first.