Filebeat cpu usage is too high

Elastic Stack Beats
1

os:CentOS release 6.5 (Final)
filebeat version:filebeat-5.6.0-1.x86_64
32 core cpu.

I have use filebeats to collect nginx logs(my log is json format) and sent to kafka. but cpu usage is too high ,but I don't know what's wrong with that。
this is my filebeat config file

filebeat.prospectors:
- input_type: log
  paths:
    - /data/logs/nginx/access.log
  document_type: nginxAccess
  tail_files: false
processors:
 - decode_json_fields:
     fields: ["message",]
     process_array: false
     max_depth: 1
     target:
     overwrite_keys: false

output.kafka:
  hosts: ["kafa1:9092", "kafka2:9092", "kafa3:9092","kafka4:9092","kafka5:9092"]
  topic: '%{[type]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

after I start filebeat ,I type top to monitor the usage of cpu ,it mount to 1000%+ like this

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                          
121961 root      20   0  143m 136m 5500 S 1129.9  0.1 108:59.94 filebeat

plz tell me how to reduce cpu usage！
I would be very grateful

2

maybe It was caused by kafka's version，Filebeat is incompatible with kafka。

3

Which Kafka version do you have? Filebeat should be compatible with most versions.

While using that much CPU, how many lines does it process and send to kafka? You can view that in the metrics printed every 30s in the logs.

5

My kafka version is 2.11-0.11.0.0,but I found the website show the filebeat is compatible with 0.8，0.9，0.10。 so I Reduce the version to kafka 0.10。 cpu usage become 600+%
like this:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
155476 root 20 0 87964 72m 5488 S 652.5 0.1 3:47.88 filebeat

my filebeat log like this:
2017-09-21T09:07:02+08:00 INFO Non-zero metrics in the last 30s: libbeat.kafka.call_count.PublishEvents=102 libbeat.kafka.published_and_acked_events=210944 libbeat.output.kafka.bytes_read=1199168 libbeat.output.kafka.bytes_write=41040142 libbeat.publisher.published_events=208896 publish.e

6

This topic was automatically closed after 21 days. New replies are no longer allowed.