# Performance Issues with ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [August 12, 2015, 4:13pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273 "2015-08-12T16:13:48Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Igor\_Zeiger](https://avatars.discourse-cdn.com/v4/letter/i/9fc29f/32.png) [@Igor\_Zeiger](https://discuss.elastic.co/u/Igor_Zeiger)\ **Post date:** [August 12, 2015, 4:13pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/1 "2015-08-12T16:13:48Z")
Hi, We're currently facing some performance issues with our ElastiSearch cluster and trying to find what is the issue and how we may solve it. We have system with Nxlog -\> Logstash Broker -\> Redis -\> 12 Logstash clients -\> 9 ElasticSearch Node + 1 ElasticSearch Master. At some point we hit the situation when the data processing slows down. The symptoms are that logstash machine do not take data with the same speed, broker puts it to Redis, which causes two things: 1 - delay in putting data to shards - we can see up to couple of hours in data processing 2 - redis queue became overloaded, reaching up to 30 million documents and redis just being killed by OS We don't see any specific metrics in Marvel or/and HQ o KOPF plugins that ES nodes are overloaded, everything looks absolutely normal. So, I appreciate any help or advice, since we don't see anything that can help us identify the problem Below is our configuration: Logstash Broker: input { file { type =\> "syslog\_product" path =\> ["/data/product/\*"] sincedb\_path =\> "/data/sincedb" } } output { stdout {} redis { host =\> ["euwest-redis"] data\_type =\> "list" key =\> "product:syslog\_product" type =\> "syslog\_product" batch =\> true workers =\> 8 } } Logstash Machines: input { redis { ``` host => ["euwest-redis"] data_type => "list" key => "product:syslog_product" type => "syslog_product" tags => "product_pri" threads => 8 batch_count => 200 ``` } } filter { grok { match =\> ["message", "%{DATA:hostname} %{DATA:cluster} %{GREEDYDATA:empty} - - - [%{MONTHDAY:day}/%{MONTH:month}/%{YEAR:year}:%{HOUR:hour}:%{MINUTE:minute}:%{SECOND :second}+%{GREEDYDATA:empty}] {{ %{DATA:http\_request} /%{DATA:snippet}/%{DATA:referer} }} %{DATA:http\_code} {{ %{DATA:empty} }} {{ %{DATA:url} }} {{ %{DATA:browser} }} {{ %{DATA:empty} }} {{ %{DATA:client\_ip} }} {{ %{DATA:empty} {{ %{DATA:empty} }} {{ %{DATA:empty} }} {{ %{DATA:empty} }} {{ %{DATA:session\_time} }} {{ %{DATA:empty} }} {{ %{DATA:session\_id} }} {{ %{DATA:snippet\_id} }} {{ %{DATA:product\_version} }} {{ %{DATA:papyrus\_revision} }}"] } mutate { replace =\> ["@source\_host", "%{hostname}"] remove =\> ["empty", "@source\_path", "@source"] convert =\> ["snippet", "integer", "session\_time", "float"] } date { match =\> ["MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"] } if "\_grokparsefailure" in [tags] { drop {} } } output { elasticsearch { cluster =\> "G177" host =\> "euwest-elastic" port =\> "9300" index =\> "logstash-%{+YYYY.MM.dd}" manage\_template =\> false } } ElasticSearch Node : cluster.name: G177 node.name: elasticsearch-euwest-qqqq node.master: false node.data: true bootstrap.mlockall: true discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["MASTER IP"] network.host: _eth0:ipv4_ path.conf: /etc/elasticsearch path.data: /ebs/elasticsearch path.logs: /data/logs/elasticsearch path.plugins: /usr/share/elasticsearch/plugins indices.memory.index\_buffer\_size: 50% index.translog.flush\_threshold\_ops: 50000 index.store.type: mmapfs index.refresh\_interval: 10s indices.fielddata.cache.size: 25% indices.cluster.send\_refresh\_mapping: false index.number\_of\_replicas: 1 index.search.slowlog.threshold.query.warn: 10s index.search.slowlog.threshold.query.info: 5s index.search.slowlog.threshold.query.debug: 2s index.search.slowlog.threshold.query.trace: 500ms index.search.slowlog.threshold.fetch.warn: 1s index.search.slowlog.threshold.fetch.info: 800ms index.search.slowlog.threshold.fetch.debug: 500ms index.search.slowlog.threshold.fetch.trace: 200ms index.indexing.slowlog.threshold.index.warn: 10s index.indexing.slowlog.threshold.index.info: 5s index.indexing.slowlog.threshold.index.debug: 2s index.indexing.slowlog.threshold.index.trace: 500ms indices.store.throttle.type: none ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 12, 2015, 10:55pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/2 "2015-08-12T22:55:37Z")
What versions are you on? ---
**Author:** ![Igor\_Zeiger](https://avatars.discourse-cdn.com/v4/letter/i/9fc29f/32.png) [@Igor\_Zeiger](https://discuss.elastic.co/u/Igor_Zeiger)\ **Post date:** [August 13, 2015, 6:51am UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/3 "2015-08-13T06:51:56Z")
Logstash: 1.5.3 ElasticSearch: Version: 1.4.4, Build: c88f77f/2015-02-19T13:05:36Z, JVM: 1.7.0\_79 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 13, 2015, 7:27am UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/4 "2015-08-13T07:27:36Z")
How much data in the cluster? How many nodes and what are their specs? I'd suggest upgrading ES (irrespective of those answers). ---
**Author:** ![Igor\_Zeiger](https://avatars.discourse-cdn.com/v4/letter/i/9fc29f/32.png) [@Igor\_Zeiger](https://discuss.elastic.co/u/Igor_Zeiger)\ **Post date:** [August 14, 2015, 9:09pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/5 "2015-08-14T21:09:26Z")
We have 1 Master Node, 2 Search Nodes and 9 Data Nodes. We store 40 days of data. Each day is about 500Gb. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 14, 2015, 10:06pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/6 "2015-08-14T22:06:42Z")
How much RAM and heap for the data nodes? ---
**Author:** ![Igor\_Zeiger](https://avatars.discourse-cdn.com/v4/letter/i/9fc29f/32.png) [@Igor\_Zeiger](https://discuss.elastic.co/u/Igor_Zeiger)\ **Post date:** [August 17, 2015, 6:55pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/7 "2015-08-17T18:55:20Z")
Each data node has 30Gb of Memory. The heap size is set to 25Gb : usr/bin/java -Xms25g -Xmx25g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-0.90.9.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 17, 2015, 9:56pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/8 "2015-08-17T21:56:36Z")
That my be part of if then. We recommend setting heap to 50% of total system memory to allow the OS to cache the underlying lucene files to help performance. ---
**Author:** ![Igor\_Zeiger](https://avatars.discourse-cdn.com/v4/letter/i/9fc29f/32.png) [@Igor\_Zeiger](https://discuss.elastic.co/u/Igor_Zeiger)\ **Post date:** [August 18, 2015, 1:01pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/9 "2015-08-18T13:01:06Z")
Thanks! But if I reduce heap size to 15Gb, wouldn't it create problems with java memory. We had some issues, when heap was reaching 100%, causing Java to crash. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 18, 2015, 10:03pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/10 "2015-08-18T22:03:16Z")
Then your cluster is overloaded and you need more resources or nodes, or less data. There's only so much you can do with a given set of resources 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:55pm UTC](https://discuss.elastic.co/t/performance-issues-with-elasticsearch/27273/11 "2017-07-05T23:55:10Z")