I'm using the ELK stack to receive and parse logfiles from an Apache server. My Elasticsearch cluster consists of 3 nodes, with Logstash running on one of these nodes.
I've set up Filebeat on the machine that stores the logfiles as they're produced and have it pointing at my Logstash instance.
Previously, before using Filebeat I had no problems parsing logs directly from Logstash to Elasticsearch using the "file" input plugin, however, since beginning to use Filebeat, my indexing rate in Elasticsearch and events received/emitted rate in Logstash has plummeted.
This is a sample of what my events received and emitted over time looks like in Logstash (captured via X-Pack Monitoring):
Logstash begins receiving an acceptable amount of throughput from Filebeat but quickly slows down to almost non-existant rates (and eventually stops). At the same time, event latency is also increasing.
For reference, I will include both my Logstash.yml and Filebeat.yml configuration.
Logstash.yml
path.data: /var/lib/logstash
path.logs: /var/logs/logstash
path.settings: /etc/logstash
pipeline.workers: 4
pipeline.batch.size: 1000
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://hostname:9200
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "changeme"
log.level: info
Filebeat.yml
filebeat.spool_size: 4000
filebeat.idle_timeout: 5s
filebeat.prospectors:
- input_type: log
paths:
- /log/file/path/access_log-*
close_inactive: 30s
harvester_limit: 1024
output.logstash:
hosts: ["hostname:5045"]
bulk_max_size: 1000
workers: 4
logging.level: debug
logging.selectors: ["*"]
I'm finding it tough to diagnose exactly what is causing the slowdown as the logs move from Filebeat > Logstash > Elasticsearch. Any and all help appreciated. I can also post any other configuration files that might be helpful.