Filebeat OOM

Elastic Stack Beats
1

Hi, running filebeat 6.2.2

Today I noticed filebeat was going bonkers... In /var/log/messages I see allot of these...

Aug 30 13:30:43 master-0002 kernel: Out of memory: Kill process 19154 (filebeat) score 818 or sacrifice child
Aug 30 13:31:40 master-0002 filebeat: fatal error: runtime: out of memory

The metrics output (as you can see it's not monitoring many files:

2018-08-30T15:37:15.442Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31800,"time":31803},"total":{"ticks":58140,"time":58143,"value":58140},"user":{"ticks":26340,"time":26340}},"info":{"ephemeral_id":"2e507519-2788-4e40-b4e4-3964f97af9c5","uptime":{"ms":38552}},"memstats":{"gc_next":13559486432,"memory_alloc":9792227856,"memory_total":21843088664,"rss":15713218560}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"kafka"},"pipeline":{"clients":2,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":2},"system":{"cpu":{"cores":4},"load":{"1":0.66,"15":0.27,"5":0.36,"norm":{"1":0.165,"15":0.0675,"5":0.09}}}}}}

The config:

filebeat.prospectors:
- type: log
  paths:
    - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stdout*
    - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stderr*
  exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
  fields:
    source_type: "framework"
  fields_under_root: true
  tail_files: false

- type: log
  paths:
    - /var/log/mesos/*.log
    - /var/log/dcos/dcos.log
  fields:
    source_type: "dcos"
  fields_under_root: true
  tail_files: false

output.kafka:
  hosts: ["xxxxxx:9092"]

  version: 0.11.0.0

  topic: "app-logs"
  partition.round_robin:
    reachable_only: true

  required_acks: -1
  max_message_bytes: 2000000
2

@javadevmtl

You are right, by looking at the metric output you only have 1 file open, if you look at your filebeat log is there any stacktrace present? Is Filebeat generating any errors in the log?

I see that you are running 6.2.2, I know that we have fixed a few leaks in 6.3 and also in 6.4 did you try with the latest release of filebeat if you are running into the same issue?

3

Unfortunately I didn't catch the log while it happened and I restarted since then...

4

Well I'm running Elastic 6.2.2. Can I run a newer filebeat? We are using kafka as the queue between the 2.

5

@pierhugues I'm assuming I can use 6.4.0 Filebeat with Elastic 6.2.4 since I have Kafka in the middle?

6

Yes you should be able to use it in that scenario.

7

Ok I will have to test and eventually deploy it... Thanks.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.