Logstash Event Processing is decreasing as times go by


(Archelle Pagapulan) #1

Here's what we did:

  • Started the logstash.
  • After few hours, logstash performance is going down and little by little, few events are being processed.
  • We checked the metricbeat report before restarting the logstash and found out that there is high IO weight on our logstash servers.
  • Restarted the logstash.
  • After restart, as expected, there is a spike in event processing, after a minute, the logstash processing becomes normal, then again, few events are being processed.
  • We also noticed that logstash doesnt load balance the CPU usage according to metricbeat report.

Few Notes:

  • We are not using persistent queues
  • No error logs are written in logstash's log directory.
  • We are using logstash version 6.1.1

logstash.yml : We leave everything as default except

pipeline.workers: 12
pipeline.output.workers: 6

logstash.conf

input {
	beats {
		port => "5044"
	}
}
filter {
	// We have used grok, kv, mutate and ruby filters here
}
output {
	elasticsearch {
		hosts => ["hostname1:9200", "hostname2:9200", "hostname3:9200"]
		index => "index-%{+YYYY.MM.dd}"
              		template_name => "template"
              }
}

(Archelle Pagapulan) #2

Looks like our logstash filter isnt optimized to handle huge data.. but dont know why logstash didnt throws error on this.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.