Filebeat slowing to a halt within 20-30 minutes of starting


#1

I'm using the ELK stack to receive and parse logfiles from an Apache server. My Elasticsearch cluster consists of 3 nodes, with Logstash running on one of these nodes.

I've set up Filebeat on the machine that stores the logfiles as they're produced and have it pointing at my Logstash instance.

Previously, before using Filebeat I had no problems parsing logs directly from Logstash to Elasticsearch using the "file" input plugin, however, since beginning to use Filebeat, my indexing rate in Elasticsearch and events received/emitted rate in Logstash has plummeted.

This is a sample of what my events received and emitted over time looks like in Logstash (captured via X-Pack Monitoring):

Logstash begins receiving an acceptable amount of throughput from Filebeat but quickly slows down to almost non-existant rates (and eventually stops). At the same time, event latency is also increasing.

For reference, I will include both my Logstash.yml and Filebeat.yml configuration.

Logstash.yml

path.data: /var/lib/logstash   
path.logs: /var/logs/logstash
path.settings: /etc/logstash

pipeline.workers: 4
pipeline.batch.size: 1000

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://hostname:9200
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password:  "changeme"

log.level: info

Filebeat.yml

filebeat.spool_size: 4000
filebeat.idle_timeout: 5s


filebeat.prospectors:
- input_type: log
  paths:
     - /log/file/path/access_log-*

  close_inactive: 30s
  harvester_limit: 1024

output.logstash:
  hosts: ["hostname:5045"]
  bulk_max_size: 1000
  workers: 4

logging.level: debug
logging.selectors: ["*"]

I'm finding it tough to diagnose exactly what is causing the slowdown as the logs move from Filebeat > Logstash > Elasticsearch. Any and all help appreciated. I can also post any other configuration files that might be helpful.


(Steffen Siering) #2

can you share versions and logs?


#3

I'm running the latest version of each part of the stack, so 5.3 I believe?


(Steffen Siering) #4

And logs?


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.