# Logstash pusher performance: 2.2.4 vs 5.0.2

**URL:** <https://discuss.elastic.co/t/logstash-pusher-performance-2-2-4-vs-5-0-2/68481>\
**Category:** Logstash\
**Created:** [December 8, 2016, 8:33pm UTC](https://discuss.elastic.co/t/logstash-pusher-performance-2-2-4-vs-5-0-2/68481 "2016-12-08T20:33:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bevan](https://avatars.discourse-cdn.com/v4/letter/b/9fc29f/32.png) [@Bevan](https://discuss.elastic.co/u/Bevan)\
**Post date:** [December 8, 2016, 8:33pm UTC](https://discuss.elastic.co/t/logstash-pusher-performance-2-2-4-vs-5-0-2/68481/1 "2016-12-08T20:33:31Z")

</div>

We have a fairly active ELK stack set up with the following configuration:

1. Local pushers running on the application servers with:

- file input using multiline codec over multiple logfiles
- minimal filter adding host-specific fields to the message
- trim filter truncating any message larger than X bytes
- s3 output

1. Indexers in AWS doing the heavy grokking with s3 input and es output

Several servers push around 50,000 lines per minute, but our "batch processing servers" can generate up to 250,000 lines per minute.  
The trim filter is needed to prevent overly large log lines from causing S3 upload failures.

When we tried to upgrade these pushers to 5.0 (using the same config) throughput on the application servers dropped to around 4k/minute and throughput on the batchservers dropped to around 13k/minute. In both cases around a factor of 20 difference from what we were getting before.

We tried a few general modifications, but were unable to get the performance we need and had to roll back to 2.2.4.  
Some things we tried:

- Giving every logfile on a server it's own input section ([https://www.elastic.co/guide/en/logstash/current/execution-model.html](https://www.elastic.co/guide/en/logstash/current/execution-model.html) suggested each input section gets it's own thread).
- Switching the trim filter from a ruby code section to a regexp grok mutate (the old code ran into some deprecation issues)

Has anyone else had any similar experiences with 5.0 pusher performance, or, even better, managed to get their pushers pushing faster?  
Does anyone have any suggestions for other tuning we could try or infrastructure changes we could make?  
I'd like to move forward with the new logstash if possible...

---

<div class="post-metadata">

**Author:** ![Bevan](https://avatars.discourse-cdn.com/v4/letter/b/9fc29f/32.png) [@Bevan](https://discuss.elastic.co/u/Bevan)\
**Post date:** [December 14, 2016, 6:34pm UTC](https://discuss.elastic.co/t/logstash-pusher-performance-2-2-4-vs-5-0-2/68481/2 "2016-12-14T18:34:06Z")

</div>

Does anyone at least have some logstash pusher metrics to share?  
Are we asking for a reasonable volume or is this something unusual?  
If unusual, what are people doing differently to avoid this bottleneck in 5.0?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2017, 6:34pm UTC](https://discuss.elastic.co/t/logstash-pusher-performance-2-2-4-vs-5-0-2/68481/3 "2017-01-11T18:34:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.