Logstash Event Processing is decreasing as times go by

Here's what we did:

  • Started the logstash.
  • After few hours, logstash performance is going down and little by little, few events are being processed.
  • We checked the metricbeat report before restarting the logstash and found out that there is high IO weight on our logstash servers.
  • Restarted the logstash.
  • After restart, as expected, there is a spike in event processing, after a minute, the logstash processing becomes normal, then again, few events are being processed.
  • We also noticed that logstash doesnt load balance the CPU usage according to metricbeat report.

Few Notes:

  • We are not using persistent queues
  • No error logs are written in logstash's log directory.
  • We are using logstash version 6.1.1

logstash.yml : We leave everything as default except

pipeline.workers: 12
pipeline.output.workers: 6


input {
	beats {
		port => "5044"
filter {
	// We have used grok, kv, mutate and ruby filters here
output {
	elasticsearch {
		hosts => ["hostname1:9200", "hostname2:9200", "hostname3:9200"]
		index => "index-%{+YYYY.MM.dd}"
              		template_name => "template"

Looks like our logstash filter isnt optimized to handle huge data.. but dont know why logstash didnt throws error on this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.