# Logstash Event Processing is decreasing as times go by

**URL:** <https://discuss.elastic.co/t/logstash-event-processing-is-decreasing-as-times-go-by/133963>\
**Category:** Logstash\
**Created:** [May 31, 2018, 4:21am UTC](https://discuss.elastic.co/t/logstash-event-processing-is-decreasing-as-times-go-by/133963 "2018-05-31T04:21:06Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\
**Post date:** [May 31, 2018, 4:21am UTC](https://discuss.elastic.co/t/logstash-event-processing-is-decreasing-as-times-go-by/133963/1 "2018-05-31T04:21:07Z")

</div>

Here's what we did:

- Started the logstash.
- After few hours, logstash performance is going down and little by little, few events are being processed.
- We checked the metricbeat report before restarting the logstash and found out that there is high IO weight on our logstash servers.
- Restarted the logstash.
- After restart, as expected, there is a spike in event processing, after a minute, the logstash processing becomes normal, then again, few events are being processed.
- We also noticed that logstash doesnt load balance the CPU usage according to metricbeat report.

Few Notes:

- We are not using persistent queues
- No error logs are written in logstash's log directory.
- We are using logstash version 6.1.1

logstash.yml : We leave everything as default except

```auto
pipeline.workers: 12
pipeline.output.workers: 6

```

logstash.conf

```auto
input {
	beats {
		port => "5044"
	}
}
filter {
	// We have used grok, kv, mutate and ruby filters here
}
output {
	elasticsearch {
		hosts => ["hostname1:9200", "hostname2:9200", "hostname3:9200"]
		index => "index-%{+YYYY.MM.dd}"
              		template_name => "template"
              }
}

```

---

<div class="post-metadata">

**Author:** ![inhinyera16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inhinyera16/32/61625_2.png) [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\
**Post date:** [June 5, 2018, 5:20am UTC](https://discuss.elastic.co/t/logstash-event-processing-is-decreasing-as-times-go-by/133963/2 "2018-06-05T05:20:59Z")

</div>

Looks like our logstash filter isnt optimized to handle huge data.. but dont know why logstash didnt throws error on this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 3, 2018, 5:21am UTC](https://discuss.elastic.co/t/logstash-event-processing-is-decreasing-as-times-go-by/133963/3 "2018-07-03T05:21:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.