# Performance degradation after upgrading to ES 2.2.0

**URL:** <https://discuss.elastic.co/t/performance-degradation-after-upgrading-to-es-2-2-0/43396>\
**Category:** Elasticsearch\
**Created:** [March 3, 2016, 2:31pm UTC](https://discuss.elastic.co/t/performance-degradation-after-upgrading-to-es-2-2-0/43396 "2016-03-03T14:31:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Seva\_Feldman](https://avatars.discourse-cdn.com/v4/letter/s/fbc32d/32.png) [@Seva\_Feldman](https://discuss.elastic.co/u/Seva_Feldman)\
**Post date:** [March 3, 2016, 2:31pm UTC](https://discuss.elastic.co/t/performance-degradation-after-upgrading-to-es-2-2-0/43396/1 "2016-03-03T14:31:30Z")

</div>

Hi Folks,

I've created new cluster based on ES 2.2.0, Logstash 2.2.0. Logstash input kafka and output elasticsearch.  
Cluster built on top of AWS 10 data nodes r3.2xlarge with 3TB data\_dir and 12 logstash indexers with local elasticsearch (no\_data and no\_master) on c3.large instances. All app server are producing logs to kafka and indexers are consuming them into ES. ES cluster indexing messages about 20% slower than app server are pushing them to kafka. Daily EC cluster receiving about 1.5 billion messages.  
Data node load is about 30%, iowait is 15%, io util is about 30%-50%.  
Same setup on ES 1.7 worked well.

Logstash conf:  
\*\* input {\*\*  
\*\* kafka {\*\*  
\*\* topic\_id =\> "apache-accesslog"\*\*  
\*\* zk\_connect =\> "zookeeper1a:2181,zookeeper1b:2181,zookeeper1c:2181/kafka"\*\*  
\*\* queue\_size =\> 3000\*\*  
\*\* group\_id =\> "logstash\_indexer\_apache\_accesslog"\*\*  
\*\* }\*\*  
\*\* }\*\*  
\*\* filter {\*\*  
\*\* if "metric" in [tags] or "\_bad\_clientip" in [tags] {\*\*  
\*\* drop {}\*\*  
\*\* }\*\*  
\*\* }\*\*  
\*\* output {\*\*  
\*\* if "apache-accesslog" in [tags] {\*\*  
\*\* elasticsearch {\*\*  
\*\* hosts =\> ["localhost:9200"]\*\*  
\*\* index =\> "apache-accesslog-%{+YYYY.MM.dd}"\*\*  
\*\* flush\_size =\> 3000\*\*  
\*\* workers =\> 3\*\*  
\*\* }\*\*  
\*\* }\*\*  
\*\* }\*\*

Elasticsearch on logstash indexer node:

**[cluster.name](http://cluster.name): production.eu-west-1**  
**[node.name](http://node.name): logstash-i-95e52a1f.a.production.eu-west-1**  
**node.max\_local\_storage\_nodes: 1**  
**path.conf: /etc/elasticsearch**  
**path.data: /usr/share/elasticsearch**  
**path.logs: /var/log/elasticsearch**  
**network.host: _eth0:ipv4_,_local_**  
**http.port: 9200**  
**discovery.zen.ping.multicast.enabled: true**  
**action.destructive\_requires\_name: true**  
**cloud.node.auto\_attributes: true**  
**cloud.aws.access\_key: somekey**  
**cloud.aws.region: eu-west-1**  
**cloud.aws.secret\_key: somekey**  
**cluster.routing.allocation.awareness.attributes: zone**  
**cluster.routing.allocation.awareness.force.zone.values: spot,ondemand**  
**discovery.ec2.any\_group: true**  
**discovery.ec2.availability\_zones: eu-west-1a,eu-west-1b,eu-west-1c**  
**discovery.ec2.host\_type: private\_ip**  
**discovery.ec2.ping\_timeout: 10s**  
**discovery.ec2.tag.elasticsearch\_cluster: production.eu-west-1**  
**discovery.type: ec2**  
**gateway.expected\_nodes: 0**  
**node.data: false**  
**node.master: false**  
**node.zone: spot**

Logstash and ES are getting each 40% of the memory on the instance.

Kafka input are working very fast, about 6k events/s (used metrics filter to measure). However es output is very slow.

index mapping config:

**{**  
\*\* "order": 0,\*\*  
\*\* "template": "apache-accesslog-\*",\*\*  
\*\* "settings": {\*\*  
\*\* "index": {\*\*  
\*\* "routing": {\*\*  
\*\* "allocation": {\*\*  
\*\* "total\_shards\_per\_node": "4"\*\*  
\*\* }\*\*  
\*\* },\*\*  
\*\* "cache": {\*\*  
\*\* "field": {\*\*  
\*\* "type": "soft"\*\*  
\*\* },\*\*  
\*\* "filter": {\*\*  
\*\* "expire": "30m"\*\*  
\*\* }\*\*  
\*\* },\*\*  
\*\* "refresh\_interval": "30s",\*\*  
\*\* "number\_of\_shards": "8",\*\*  
\*\* "translog": {\*\*  
\*\* "flush\_threshold\_ops": "20000"\*\*  
\*\* },\*\*  
\*\* "auto\_expand\_replicas": "false",\*\*  
\*\* "query": {\*\*  
\*\* "default\_field": "uri"\*\*  
\*\* },\*\*  
\*\* "store": {\*\*  
\*\* "throttle": {\*\*  
\*\* "type": "node",\*\*  
\*\* "max\_bytes\_per\_sec": "100mb"\*\*  
\*\* }\*\*  
\*\* },\*\*  
\*\* "number\_of\_replicas": "1"\*\*  
\*\* }\*\*  
\*\* },\*\*  
\*\* "mappings": {\*\*  
\*\* "_default_": {\*\*  
\*\* "\_source": {\*\*  
\*\* "enabled": true\*\*  
\*\* },\*\*  
\*\* "dynamic\_templates": [\*\*  
\*\* {\*\*  
\*\* "integers": {\*\*  
\*\* "mapping": {\*\*  
\*\* "index": "not\_analyzed",\*\*  
\*\* "type": "integer",\*\*  
\*\* "doc\_values": true\*\*  
\*\* },\*\*  
\*\* "match\_mapping\_type": "integer"\*\*  
\*\* }\*\*  
\*\* },\*\*  
\*\* {\*\*  
\*\* "strings": {\*\*  
\*\* "mapping": {\*\*  
\*\* "index": "not\_analyzed",\*\*  
\*\* "type": "string",\*\*  
\*\* "doc\_values": true\*\*  
\*\* },\*\*  
\*\* "match\_mapping\_type": "string"\*\*  
\*\* }\*\*  
\*\* }\*\*  
\*\* ],\*\*  
\*\* "\_all": {\*\*  
\*\* "enabled": false\*\*  
\*\* }\*\*

Changes I did during investigation was adding batch\_size,workers to logstash process; workers and flush\_size to output/ queue\_size to input. Workers made it working faster, but than I started to get 429 error from logstash.

Any input or suggestions will be appreciated!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:11pm UTC](https://discuss.elastic.co/t/performance-degradation-after-upgrading-to-es-2-2-0/43396/2 "2017-07-05T23:11:35Z")

</div>