# Logstash abysmal performance from 2.2.0 to 2.3.1

**URL:** <https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638>\
**Category:** Logstash\
**Created:** [April 18, 2016, 10:01am UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638 "2016-04-18T10:01:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![KlavsKlavsen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/klavsklavsen/32/74145_2.png) [@KlavsKlavsen](https://discuss.elastic.co/u/KlavsKlavsen)\
**Post date:** [April 18, 2016, 10:01am UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638/1 "2016-04-18T10:01:23Z")

</div>

Hi guys,

We are running logstash 2.2.0 (against elasticsearch 2.3.1) and it can index \>1.2mill docs/minute.  
If we then upgrade to logstash-2.3.1 - it peaks at 200k docs/min.. 1/6 of the performance.

There's nothing in the errorlog of either elasticsearch, nor logstash.

Any ideas as to what I could try to remedy this, are very welcome.. (should I test with 5.0 alpha?)

Config is this:  
input {  
redis {  
host =\> "127.0.0.1"  
# Remember that type does NOT overwrite trype from shipper!  
type =\> "redis-input"  
# these settings should match the output of the agent  
data\_type =\> "list"  
key =\> "logstash"  
codec =\> json  
threads =\> 8  
}  
redis {  
host =\> "127.0.0.1"  
type =\> "netflow"  
data\_type =\> "list"  
key =\> "pet1year"  
codec =\> json  
threads =\> 2  
}  
redis {  
host =\> "[ytes02.example.org](http://ytes02.example.org)"  
# Remember that type does NOT overwrite trype from shipper!  
type =\> "redis-input"  
# these settings should match the output of the agent  
data\_type =\> "list"  
key =\> "logstash"  
codec =\> json  
threads =\> 8  
}  
redis {  
host =\> "[ytes02.example.org](http://ytes02.example.org)"  
type =\> "netflow"  
data\_type =\> "list"  
key =\> "pet1year"  
codec =\> json  
threads =\> 2  
}  
}

filter {  
#choose index  
#1.5.0 only feature  
if [type] == "cnrdhcp" {  
mutate { add\_field =\> { "[index]" =\> "cnrdhcp-%{+YYYY.MM.dd}" } }  
} else if [type] == "netflow" {  
mutate { add\_field =\> { "[index]" =\> "netflow-%{+YYYY.MM.dd}" } }  
} else if [type] == "akamai\_access\_logs" {  
mutate { add\_field =\> { "[index]" =\> "cdn\_access\_logs-%{+YYYY.MM.dd}" } }  
} else if [type] == "cdn\_access\_logs" {  
mutate { add\_field =\> { "[index]" =\> "cdn\_access\_logs-%{+YYYY.MM.dd}" } }  
} else if [type] == "cdn\_content\_logs" {  
mutate { add\_field =\> { "[index]" =\> "cdn\_content\_logs-%{+YYYY.MM.dd}" } }  
} else if [type] == "mpf\_arkiv" {  
mutate { add\_field =\> { "[index]" =\> "mpf\_arkiv-%{+[YYYY.MM](http://YYYY.MM)}" } }  
} else {  
mutate { add\_field =\> { "[index]" =\> "logstash-%{+YYYY.MM.dd}" } }  
}  
#generate message\_id if its not present..  
if [message\_id] {  
mutate { add\_tag =\> "hasmessage\_id" }  
} else if [message] {  
ruby {  
init =\> "require 'digest/sha1'"  
code =\> "event['message\_id'] = Digest::SHA1.base64digest(event['message'])"  
}  
} else {  
#really broken input.. use timestamp as id for now.. - we should never land here  
mutate { add\_field =\> { "[@metadata][id]" =\> "%{@timestamp}" } }  
}  
}

output {  
elasticsearch {  
codec =\> plain {  
charset =\> 'UTF-8'  
}  
hosts =\> "127.0.0.1:9200"  
index =\> "%{[index]}"  
manage\_template =\> false  
document\_id =\> "%{[message\_id]}"  
}  
statsd {  
host =\> "localhost"  
port =\> 8125  
sender =\> "ytes01"  
namespace =\> "servers"  
increment =\> "logstash.processing"  
}

}

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [April 18, 2016, 2:29pm UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638/2 "2016-04-18T14:29:31Z")

</div>

In version 2.0.5 of the Redis Input we enabled batching by default (size 125). I think the multiple threads are stomping on each each other.

You can try one of:

1. removing the extra threads - allows up to 125 'events' to be fetched in one thread per input. 125 was chosen because it works well with the pipeline batch count.
2. "revert to previous behaviour" - set batch count to 1 - `batch_count => 1`

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [April 18, 2016, 2:51pm UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638/3 "2016-04-18T14:51:33Z")

</div>

I also suspect that batch\_count \> 1 with threads \> 1 maybe crashing one of the 8 inputs `(threads => 8)` and it is restarted.

This can be seen quite clearly using --debug.  
Look for:  
`A plugin had an unrecoverable error. Will restart this plugin.` with `Plugin: <LogStash::Inputs::Redis...`

---

<div class="post-metadata">

**Author:** ![KlavsKlavsen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/klavsklavsen/32/74145_2.png) [@KlavsKlavsen](https://discuss.elastic.co/u/KlavsKlavsen)\
**Post date:** [April 20, 2016, 1:45pm UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638/4 "2016-04-20T13:45:03Z")

</div>

We tried tweaking the redis input settings with no change. We then tried setting output to not putting to ES.. and everything ran quickly - which made us conclude the input wasn't the issue - it was the output.

We added workers =\> 8 to elasticsearch output - and this increased throughput to the same as with 2.2.0. We also tried higher numbers (16 or 20) with no improvements.

Thank you for the input 🙂

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [April 22, 2016, 8:28am UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638/5 "2016-04-22T08:28:58Z")

</div>

Please confirm your solution (for future readers of this discussion):  
e.g. You now have no threads config settings in the redis inputs.

You are advised to read the [upgrade to 2.2 blog post](https://www.elastic.co/blog/upgrade-guide-for-logstash-2-2) and the **New Pipeline and Outputs** section in particular.

At the time of writing this, there remains an ongoing issue the the elasticsearch output with output workers \> 1 and connections not pooled.

---

<div class="post-metadata">

**Author:** ![KlavsKlavsen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/klavsklavsen/32/74145_2.png) [@KlavsKlavsen](https://discuss.elastic.co/u/KlavsKlavsen)\
**Post date:** [April 25, 2016, 12:21pm UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638/6 "2016-04-25T12:21:23Z")

</div>

yes. No threads not batch\_count set on redis input.. workers = 8 in elasticsearch output.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:00am UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638/7 "2017-07-06T05:00:44Z")

</div>