Here's what we did:
- Started the logstash.
- After few hours, logstash performance is going down and little by little, few events are being processed.
- We checked the metricbeat report before restarting the logstash and found out that there is high IO weight on our logstash servers.
- Restarted the logstash.
- After restart, as expected, there is a spike in event processing, after a minute, the logstash processing becomes normal, then again, few events are being processed.
- We also noticed that logstash doesnt load balance the CPU usage according to metricbeat report.
Few Notes:
- We are not using persistent queues
- No error logs are written in logstash's log directory.
- We are using logstash version 6.1.1
logstash.yml : We leave everything as default except
pipeline.workers: 12
pipeline.output.workers: 6
logstash.conf
input {
beats {
port => "5044"
}
}
filter {
// We have used grok, kv, mutate and ruby filters here
}
output {
elasticsearch {
hosts => ["hostname1:9200", "hostname2:9200", "hostname3:9200"]
index => "index-%{+YYYY.MM.dd}"
template_name => "template"
}
}