Filebeat OOM error

We've been running filebeat to tail logs using the following prospector configuration

    - input_type: log
      scan_frequency: "30s"
      close_older: "24h"
      ignore_older: "15h"
      clean_inactive: "24h"
      tail_files: false
      exclude_files: ['\.gz$']
      json.message_key: log
        - /var/log/docker/highgarden/docker-process-highgarden.log

Every once in a while, i see filebeat run out of memory and the process stops. Here is a stacktrace for the same

goroutine 199734 [running]:
	/usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc42035b5b0 sp=0xc42035b5a8
runtime.mallocgc(0x3ad64000, 0x0, 0x25a7c000, 0xc4227ca000)
	/usr/local/go/src/runtime/malloc.go:670 +0x903 fp=0xc42035b650 sp=0xc42035b5b0
runtime.growslice(0x8f5780, 0xc4227ca000, 0x2f11c000, 0x2f11c000, 0x2f11c400, 0xc4227ca000, 0x79a45d, 0xc4210a0800)
	/usr/local/go/src/runtime/slice.go:126 +0x24e fp=0xc42035b6e0 sp=0xc42035b650*Buffer).doAppend(0xc420a60aa0, 0xc4210a0800, 0x400, 0x400, 0x2cdd2000, 0xffffffffffffffff, 0x400, 0x400)
	/go/src/ +0x48f fp=0xc42035b770 sp=0xc42035b6e0*Buffer).Write(0xc420a60aa0, 0xc4210a0800, 0x400, 0x400, 0xc501efa001, 0x1e2d348c, 0x2cdd2000)
	/go/src/ +0x5b fp=0xc42035b7d0 sp=0xc42035b770*Line).decode(0xc42031c310, 0x4d3ef48c, 0xc420485b69, 0x1, 0x1)
	/go/src/ +0x216 fp=0xc42035b890 sp=0xc42035b7d0*Line).advance(0xc42031c310, 0x0, 0x1)
	/go/src/ +0x1f8 fp=0xc42035b958 sp=0xc42035b890*Line).Next(0xc42031c310, 0x429efc, 0xc42035b950, 0xc42035b8f0, 0x0, 0x0, 0x0)
	/go/src/ +0x3d fp=0xc42035b9e0 sp=0xc42035b958, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40f509, ...)
	/go/src/ +0x5e fp=0xc42035ba98 sp=0xc42035b9e0*JSON).Next(0xc420cf0f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420ba0c68, ...)
	/go/src/ +0x66 fp=0xc42035bb70 sp=0xc42035ba98*StripNewline).Next(0xc420485b70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420ba0cc0, ...)
	/go/src/ +0x66 fp=0xc42035bc18 sp=0xc42035bb70*Limit).Next(0xc420cf0f20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420ba0e58, ...)
	/go/src/ +0x66 fp=0xc42035bcc0 sp=0xc42035bc18*Harvester).Harvest(0xc421ed0d00, 0xc97da0, 0xc420cf0f20)
	/go/src/ +0x24d fp=0xc42035bf68 sp=0xc42035bcc0*Prospector).startHarvester.func1(0xc42028a000, 0xc421ed0d00, 0xc97da0, 0xc420cf0f20)
	/go/src/ +0x65 fp=0xc42035bf90 sp=0xc42035bf68

The data is produced in burst mode at ~500EPS. Looks like the GC is not able to kick that eventually leads to OOM.

Which filebeat version are you using? Can you share your full config with the output part?


Sorry for the delayed reply. We are using filebeat 5.2.0 and a custom output plugin. There was an issue in the way we were shipping the data which was leading to a lot of copying and object creation. Refactoring the code to handle that cleanly fixed the problem.


This topic was automatically closed after 21 days. New replies are no longer allowed.