Hi,
I have a fresh ELK stack based on the containers from Elastic. I am using Filebeat to send logs to it. I have Filebeat monitor all files in /var/log. When it first tries to read all those files and send them to Logstash, it crashes repeatedly and uses enormous amounts of memory (6-10G depending on what is available).
It looks like it is trying to read all content into memory and then starts sending it to Logstash.
Each time it runs it loads some data into Logstash, but it doesn't keep track. So next time it runs it inserts the same messages again.
Top output just before it crashed:
top - 14:37:23 up 1:06, 3 users, load average: 1.82, 0.69, 0.57
Tasks: 193 total, 5 running, 188 sleeping, 0 stopped, 0 zombie
%Cpu(s): 37.1 us, 42.0 sy, 0.0 ni, 0.3 id, 17.9 wa, 0.0 hi, 2.7 si, 0.0 st
KiB Mem : 16268948 total, 135560 free, 16012860 used, 120528 buff/cache
KiB Swap: 1679356 total, 432792 free, 1246564 used. 87600 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25995 root 20 0 8765300 6.000g 128 S 110.4 38.7 0:29.20 filebeat
When running as a service (so under filebeat-god) it seems it keeps restarting after the out-of-memory error, leading to lots of duplicates in Elasticsearch and never being able to finish.
My config:
filebeat.prospectors:
- input_type: log
paths:
- /var/log/*
exclude_files: [".gz$"]
- input_type: log
paths:
- /var/log/httpd/*
exclude_files: [".gz$"]
- input_type: log
paths:
- /home/jboss/wildfly/standalone/log/server*
exclude_files: [".gz$"]
multiline.pattern: ^[[:space:]]+|^Caused by:|^$
multiline.negate: false
multiline.match: before
- input_type: log
paths:
- /home/jboss/jboss-as/standalone/log/server*
exclude_files: [".gz$"]
multiline.pattern: '^[0-9]{2}\:' # All lines starting with 2 numbers, followed by a : so matching 10:.
multiline.negate: true
multiline.match: after
filebeat.spool_size: 2048
filebeat.publish_async: true
output.logstash:
hosts: ["logging:5044"]
logging.level: info
logging.to_files: true
logging.files:
rotateeverybytes: 104857600 # = 100MB
keepfiles: 7
Running it with debug output shows the following:
[root@logging ~]# /usr/share/filebeat/bin/filebeat -e -v -d "*" -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
2016/11/28 13:36:55.116950 beat.go:264: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2016/11/28 13:36:55.116988 beat.go:174: INFO Setup Beat: filebeat; Version: 5.0.1
2016/11/28 13:36:55.117000 processor.go:43: DBG Processors:
2016/11/28 13:36:55.117010 beat.go:180: DBG Initializing output plugins
2016/11/28 13:36:55.117021 logp.go:219: INFO Metrics logging every 30s
<snip>
2016/11/28 13:36:55.127228 prospector_log.go:79: DBG Prospector states cleaned up. Before: 42, After: 42
2016/11/28 13:36:55.171552 spooler.go:118: DBG Flushing spooler because spooler full. Events flushed: 2048
2016/11/28 13:36:55.184512 client.go:184: DBG Publish: {
"@timestamp": "2016-11-28T13:36:55.126Z",
"beat": {
"hostname": "logging.infra.devcloud",
"name": "logging.infra.devcloud",
"version": "5.0.1"
},
"input_type": "log",
"message": "Nov 28 14:15:02 template01 polkitd[877]: Registered Authentication Agent for unix-process:25329:262281 (system bus name :1.59 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)",
"offset": 35686,
"source": "/var/log/secure",
"type": "log"
}
<snip 2046 entries>]
2016/11/28 13:37:26.590023 client.go:184: DBG Publish: {
"@timestamp": "2016-11-28T13:37:00.699Z",
"beat": {
"hostname": "logging.infra.devcloud",
"name": "logging.infra.devcloud",
"version": "5.0.1"
},
"input_type": "log",
"message": "Sep 15 01:36:47 localhost dbus[634]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'",
"offset": 2374720,
"source": "/var/log/messages",
"type": "log"
}
fatal error: **runtime: out of memory**
runtime stack:
runtime.throw(0x980692, 0x16)
/usr/local/go/src/runtime/panic.go:566 +0x95
I am using the latest versions so
[root@logging ~]# /usr/share/filebeat/bin/filebeat -version
filebeat version 5.0.1 (amd64), libbeat 5.0.1
[root@logging ~]# docker exec logstash env|grep LOGSTASH_VERSION
LOGSTASH_VERSION=1:5.0.1-1