Filebeat.exe using over 1GB of Memory on Windows

[BrianBeaulieu]

Version 5.6.3
Happened with older 5.x versions as well.

Can't attach memory profile text file. Summary at the bottom.

Configuration is pretty simple. 3 log files that do not rotate often.
After running for 5 minutes ti goes from 21GB to 1.7GB in some cases.

Config File

filebeat.prospectors:

• input_type: log
  paths:
  • C:\ProgramData\Blah Technologies, Inc\Data Services*\logs\log-file-json
    json.message_key: log
    force_close_files: true
    fields:
    log_type: dataservices

fields:
client: ${CLIENT_CODE:set-client-code}

output.logstash:
hosts: ["monitor.blah.com:5044"]
index: "dataservices"

---

runtime.MemStats

Alloc = 264602440

TotalAlloc = 9738383304

Sys = 2225985784

Lookups = 3127

Mallocs = 529040

Frees = 521167

HeapAlloc = 264602440

HeapSys = 2152857600

HeapIdle = 1887404032

HeapInuse = 265453568

HeapReleased = 0

HeapObjects = 7873

Stack = 262144 / 262144

MSpan = 31200 / 901120

MCache = 2336 / 16384

BuckHashSys = 1497959

NextGC = 507408872

PauseNs = [1001700 998200 1003100 1004200 0 0 1000300 1000300 2000800 0 2000200 1001200 1002400 2001200 0 998600 1001400 0 1027200 999600 1001000 1001600 1001000 1000800 1000900 0 957300 1000400 0 2001200 0 1020100 1000500 0 1001200 1000100 1000200 998600 1000900 1000000 999500 2017800 1002700 1000000 999800 1000500 2000700 1000200 0 0 1001800 0 223000300 0 1001400 1001700 0 1002500 999400 818960200 0 999200 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

NumGC = 62

DebugGC = false

[BrianBeaulieu]

Running 32bit version, it crashes eventually.
5.6.4 and 6.0.0rc2

runtime: VirtualAlloc of 207028224 bytes failed with errno=487
fatal error: runtime: cannot map pages in arena address space

[steffens (Steffen Siering)]

Please properly format your configuration files, logs, and other text output using the </> button.

How complex are your JSON objects? Like, how 'long' are lines with JSON objects and how many keys do they have? Filebeat buffers quite a many events. You might consider to reduce the filebeat.spool_size to reduce the number of events in memory and/or move the actual JSON parsing to Logstash.

[BrianBeaulieu]

I'll answer when I get to work but just to update you I ran 6.0.0-rc2 overnight without high memory..

The JSON isn't really complex.. 2 levels deep.
Not super long

When i tested 6.0 originally I think I goofed it up and was running the older version.

[BrianBeaulieu]

I'm also running it in the foreground and not as a service currently