I'm using the ELK stack to receive and parse logfiles from an Apache server. My Elasticsearch cluster consists of 3 nodes, with Logstash running on one of these nodes.
I've set up Filebeat on the machine that stores the logfiles as they're produced and have it pointing at my Logstash instance.
Previously, before using Filebeat I had no problems parsing logs directly from Logstash to Elasticsearch using the "file" input plugin, however, since beginning to use Filebeat, my indexing rate in Elasticsearch and events received/emitted rate in Logstash has plummeted.
This is a sample of what my events received and emitted over time looks like in Logstash (captured via X-Pack Monitoring):
Logstash begins receiving an acceptable amount of throughput from Filebeat but quickly slows down to almost non-existant rates (and eventually stops). At the same time, event latency is also increasing.
For reference, I will include both my Logstash.yml and Filebeat.yml configuration.
path.data: /var/lib/logstash path.logs: /var/logs/logstash path.settings: /etc/logstash pipeline.workers: 4 pipeline.batch.size: 1000 xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.url: http://hostname:9200 xpack.monitoring.elasticsearch.username: "logstash_system" xpack.monitoring.elasticsearch.password: "changeme" log.level: info
filebeat.spool_size: 4000 filebeat.idle_timeout: 5s filebeat.prospectors: - input_type: log paths: - /log/file/path/access_log-* close_inactive: 30s harvester_limit: 1024 output.logstash: hosts: ["hostname:5045"] bulk_max_size: 1000 workers: 4 logging.level: debug logging.selectors: ["*"]
I'm finding it tough to diagnose exactly what is causing the slowdown as the logs move from Filebeat > Logstash > Elasticsearch. Any and all help appreciated. I can also post any other configuration files that might be helpful.