# Elastic Search Random Node High Load **URL:** \ **Category:** Elasticsearch\ **Created:** [June 20, 2013, 11:34pm UTC](https://discuss.elastic.co/t/elastic-search-random-node-high-load/12499 "2013-06-20T23:34:50Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![djovic](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@djovic](https://discuss.elastic.co/u/djovic)\ **Post date:** [June 20, 2013, 11:34pm UTC](https://discuss.elastic.co/t/elastic-search-random-node-high-load/12499/1 "2013-06-20T23:34:50Z")
Issue previously opened here: > Re-posted on the mailing list, as requested: We have a problem with our Elastic Search cluster in every environment. The cluster will sometimes get into an unstable state, wherein certain ES nodes will have load that is several times greater than the load on other nodes. This can be reproduced every time very quickly by hitting the cluster with about 40 concurrent threads while running data indexing, and much less quickly by simply running constant searches. Our cluster setup: - 4 no-data client nodes that service search requests - 1 no-data client node that sends data indexing requests. - 10 data nodes that are called by the 5 for indexing and searching. We have verified that all the boxes have: - the same hardware - CPU Intel Xeon X5550 (2.67GHz, 64-bit, 16 core) - 96GB high-end physical RAM (64GB JVM heap, 20 GB mem baseline). - the same OS - Red Hat Enterprise Linux Server release 5.9 (Tikanga) - Linux version 2.6.18-348.1.1.el5 x86\_64 - the same JVM - Sun/Oracle 1.7.0\_09 (64-bit) - the same ES version - Currently on 0.20.4 - Problem existed since 0.19.8, when we started with ES - no other software running - the same number of shards - 1 product shard per node - 13M product documents - (product schema has thousands of fields). - 1 entity shard per node - 135M entity documents - (dozens of entity data types, each with several fields). - approximately the same amount of data - product index is 25GB on disk. - entity index is 12GB on disk. - All data loads into about 20GB baseline RAM. - the exact same logical configuration { product: { settings: { index.translog.flush\_threshold\_size: 500mb index.refresh\_interval: 30s index.number\_of\_replicas: 1 index.translog.disable\_flush: false index.version.created: 190999 index.number\_of\_shards: 5 index.routing.allocation.total\_shards\_per\_node: 1 index.translog.flush\_threshold\_period: 60m index.translog.flush\_threshold\_ops: 5000 } } entity: { settings: { index.translog.flush\_threshold\_size: 500mb index.refresh\_interval: 30s index.number\_of\_replicas: 1 index.translog.disable\_flush: false index.version.created: 190999 index.number\_of\_shards: 5 index.routing.allocation.total\_shards\_per\_node: 1 index.translog.flush\_threshold\_period: 60m index.translog.flush\_threshold\_ops: 5000 } } } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 23, 2013, 1:25pm UTC](https://discuss.elastic.co/t/elastic-search-random-node-high-load/12499/2 "2013-06-23T13:25:27Z")
Hi, I took a look at the ZIP, but there is only a load graph there. Are you graphing all other ES metrics? If so, look at them and see which ones are different. If not, see my signature. If all hardware, etc. specs are the same, something is going to be causing that load and the load is just a symptom. We had a client the other day whose load would go up every few hours. The query rate was mostly continuous. So why would the load (and query latency) go up!? It turned out the proportion of the more expensive queries was higher during those high load/latency times. But before that we looked at all the metrics and eliminated everything else we could eliminate. 馃檪 ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, June 20, 2013 7:34:50 PM UTC-4, djovic wrote: > Issue previously opened here: > [Elastic Search Random Node High Load 路 Issue #3208 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/3208) > Re-posted on the mailing list, as requested: > > We have a problem with our Elastic Search cluster in every environment. > The cluster will sometimes get into an unstable state, wherein certain ES > nodes will have load that is several times greater than the load on other > nodes. > > This can be reproduced every time very quickly by hitting the cluster with > about 40 concurrent threads while running data indexing, and much less > quickly by simply running constant searches. > > Our cluster setup: > > - 4 no-data client nodes that service search requests > - 1 no-data client node that sends data indexing requests. > - 10 data nodes that are called by the 5 for indexing and searching. > > We have verified that all the boxes have: > > - the same hardware > - CPU Intel Xeon X5550 (2.67GHz, 64-bit, 16 core) > - 96GB high-end physical RAM (64GB JVM heap, 20 GB mem baseline). > - the same OS > - Red Hat Enterprise Linux Server release 5.9 (Tikanga) > - Linux version 2.6.18-348.1.1.el5 x86\_64 > - the same JVM > - Sun/Oracle 1.7.0\_09 (64-bit) > - the same ES version > - Currently on 0.20.4 > - Problem existed since 0.19.8, when we started with ES > - no other software running > - the same number of shards > - 1 product shard per node > - 13M product documents > - (product schema has thousands of fields). > - 1 entity shard per node > - 135M entity documents > - (dozens of entity data types, each with several fields). > - approximately the same amount of data > - product index is 25GB on disk. > - entity index is 12GB on disk. > - All data loads into about 20GB baseline RAM. > - the exact same logical configuration > { > product: { > settings: { > index.translog.flush\_threshold\_size: 500mb > index.refresh\_interval: 30s > index.number\_of\_replicas: 1 > index.translog.disable\_flush: false > index.version.created: 190999 > index.number\_of\_shards: 5 > index.routing.allocation.total\_shards\_per\_node: 1 > index.translog.flush\_threshold\_period: 60m > index.translog.flush\_threshold\_ops: 5000 > } > } > entity: { > settings: { > index.translog.flush\_threshold\_size: 500mb > index.refresh\_interval: 30s > index.number\_of\_replicas: 1 > index.translog.disable\_flush: false > index.version.created: 190999 > index.number\_of\_shards: 5 > index.routing.allocation.total\_shards\_per\_node: 1 > index.translog.flush\_threshold\_period: 60m > index.translog.flush\_threshold\_ops: 5000 > } > } > } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![djovic](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@djovic](https://discuss.elastic.co/u/djovic)\ **Post date:** [September 16, 2013, 11:36pm UTC](https://discuss.elastic.co/t/elastic-search-random-node-high-load/12499/3 "2013-09-16T23:36:25Z")
This problem was resolved with the upgrade to 0.90.x. DJ On Thursday, June 20, 2013 4:34:50 PM UTC-7, djovic wrote: > Issue previously opened here: > [Elastic Search Random Node High Load 路 Issue #3208 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/issues/3208) > Re-posted on the mailing list, as requested: > > We have a problem with our Elastic Search cluster in every environment. > The cluster will sometimes get into an unstable state, wherein certain ES > nodes will have load that is several times greater than the load on other > nodes. > > This can be reproduced every time very quickly by hitting the cluster with > about 40 concurrent threads while running data indexing, and much less > quickly by simply running constant searches. > > Our cluster setup: > > - 4 no-data client nodes that service search requests > - 1 no-data client node that sends data indexing requests. > - 10 data nodes that are called by the 5 for indexing and searching. > > We have verified that all the boxes have: > > - the same hardware > - CPU Intel Xeon X5550 (2.67GHz, 64-bit, 16 core) > - 96GB high-end physical RAM (64GB JVM heap, 20 GB mem baseline). > - the same OS > - Red Hat Enterprise Linux Server release 5.9 (Tikanga) > - Linux version 2.6.18-348.1.1.el5 x86\_64 > - the same JVM > - Sun/Oracle 1.7.0\_09 (64-bit) > - the same ES version > - Currently on 0.20.4 > - Problem existed since 0.19.8, when we started with ES > - no other software running > - the same number of shards > - 1 product shard per node > - 13M product documents > - (product schema has thousands of fields). > - 1 entity shard per node > - 135M entity documents > - (dozens of entity data types, each with several fields). > - approximately the same amount of data > - product index is 25GB on disk. > - entity index is 12GB on disk. > - All data loads into about 20GB baseline RAM. > - the exact same logical configuration > { > product: { > settings: { > index.translog.flush\_threshold\_size: 500mb > index.refresh\_interval: 30s > index.number\_of\_replicas: 1 > index.translog.disable\_flush: false > index.version.created: 190999 > index.number\_of\_shards: 5 > index.routing.allocation.total\_shards\_per\_node: 1 > index.translog.flush\_threshold\_period: 60m > index.translog.flush\_threshold\_ops: 5000 > } > } > entity: { > settings: { > index.translog.flush\_threshold\_size: 500mb > index.refresh\_interval: 30s > index.number\_of\_replicas: 1 > index.translog.disable\_flush: false > index.version.created: 190999 > index.number\_of\_shards: 5 > index.routing.allocation.total\_shards\_per\_node: 1 > index.translog.flush\_threshold\_period: 60m > index.translog.flush\_threshold\_ops: 5000 > } > } > } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:16am UTC](https://discuss.elastic.co/t/elastic-search-random-node-high-load/12499/4 "2017-07-06T02:16:10Z")