# Memory consumption and shard allocation **URL:** \ **Category:** Elasticsearch\ **Created:** [June 24, 2013, 1:21pm UTC](https://discuss.elastic.co/t/memory-consumption-and-shard-allocation/12524 "2013-06-24T13:21:58Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Vadim\_Kisselmann](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@Vadim\_Kisselmann](https://discuss.elastic.co/u/Vadim_Kisselmann)\ **Post date:** [June 24, 2013, 1:21pm UTC](https://discuss.elastic.co/t/memory-consumption-and-shard-allocation/12524/1 "2013-06-24T13:21:58Z")
Hi folks, i have a few questions about memory consumption during bulk (re-)indexing and shard allocation. We have a small cluster on the AWS, 3 nodes, 5 indices(5 shards & 1 replica each = 50 active shards) and approx. 50GB overall. Our setup: - ES 0.19.11 with java 7. - ES 4GB XMX - CPU: Xeon 2133MhZ, 8 cores (c1.xlarge see AWS: [http://aws.amazon.com/de/ec2/instance-types/#instance-details](http://aws.amazon.com/de/ec2/instance-types/#instance-details)) _Shard allocation:_ During normal operation we see that almost all the primary shards are on node 1 und node 2. Node 3 has only 2 primary shards and 14 replicas. We run many facet queries. Is it possible that all the queries are fired only on node 1/2 and only on primary shards? Because we have 90% load on these nodes. CPU load on node 1/2 is over 50-60% all the time, on node 3 less than 10%. What could be wrong here? _Memory consumption:_ During bulk reindexing with scan/scroll we ran into small "cluster overload" problems, see here from node 1: Catalina.log: [2013-06-21 17:54:23,751][INFO][monitor.jvm] [search.cloud.aws] [gc][ConcurrentMarkSweep][796536][41146] duration [5.7s], collections [1]/[6s], total [5.7s]/[2h], memory [3.4gb]-\>[3.2gb]/[3.9gb], all\_pools {[Code Cache] [12.1mb]-\>[12.1mb]/[48mb]}{[Par Eden Space] [143.1mb]-\>[16.5mb]/[532.5mb]}{[Par Survivor Space] [0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [3.2gb]-\>[3.2gb]/[3.3gb]}{[CMS Perm Gen] [37.3mb]-\>[37.3mb]/[82mb]} Nagios logs: - CMS Old Gen 99%(3.3GB),threadpool cache 100%(q2/c4/m4)###WARN### mem 81% - threadpool cache 100%(q4/c4/m4) - CMS Old Gen 99%(3.3GB)###WARN### mem 80%,jvm\_HeapMemoryUsage 93%(c3.9GB/u3.7GB/m3.9GB),threadpool search 92%(q0/c451/m486) What do you think about the logs above, especially catalina.log? We want to upgrade from c1.xlarge(8cores, 7GB RAM) to m1.xlarge(4 cores, 15GB RAM(8GB XMX for ES)) for each node and to increase the number of nodes. Does it make sense(Remember the high cpu load on node 1/2 with 8 cores)? Best regards Vadim -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [June 24, 2013, 8:46pm UTC](https://discuss.elastic.co/t/memory-consumption-and-shard-allocation/12524/2 "2013-06-24T20:46:59Z")
Regarding shard allocation, it could happen that some of your shards are hotter than others. The hashing algorithm knows nothing of data locality or your queries. You can use the Index Status API for a few metrics about the shards, but there is not much. Try using the Allocate API to move shards from the busy nodes to the less nodes (and vice-versa since the shard allocator will attempt to rebalance). If there a difference? Try changing it back. As far as memory consumption goes, the logs indicate you need more memory. 🙂 How big is your bulk load? Standard merge settings? Cheers, Ivan On Mon, Jun 24, 2013 at 6:21 AM, Vadim Kisselmann [v.kisselmann@gmail.com](mailto:v.kisselmann@gmail.com)wrote: > Hi folks, > > i have a few questions about memory consumption during bulk (re-)indexing > and shard allocation. > > We have a small cluster on the AWS, 3 nodes, 5 indices(5 shards & 1 > replica each = 50 active shards) and approx. 50GB overall. > Our setup: > > - ES 0.19.11 with java 7. > - ES 4GB XMX > - CPU: Xeon 2133MhZ, 8 cores (c1.xlarge see AWS: > [Amazon EC2 Instance-Typen – Amazon Web Services (AWS)](http://aws.amazon.com/de/ec2/instance-types/#instance-details)) > > _Shard allocation:_ > During normal operation we see that almost all the primary shards are on > node 1 und node 2. > Node 3 has only 2 primary shards and 14 replicas. We run many facet > queries. Is it possible that all the queries are fired only on node 1/2 and > only on primary shards? Because we have 90% load on these nodes. > CPU load on node 1/2 is over 50-60% all the time, on node 3 less than 10%. > What could be wrong here? > > _Memory consumption:_ > During bulk reindexing with scan/scroll we ran into small "cluster > overload" problems, see here from node 1: > > Catalina.log: > > [2013-06-21 17:54:23,751][INFO][monitor.jvm] [search.cloud.aws] > [gc][ConcurrentMarkSweep][796536][41146] duration [5.7s], collections > [1]/[6s], total [5.7s]/[2h], memory [3.4gb]-\>[3.2gb]/[3.9gb], all\_pools > {[Code Cache] [12.1mb]-\>[12.1mb]/[48mb]}{[Par Eden Space] > [143.1mb]-\>[16.5mb]/[532.5mb]}{[Par Survivor Space] > [0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [3.2gb]-\>[3.2gb]/[3.3gb]}{[CMS Perm Gen] > [37.3mb]-\>[37.3mb]/[82mb]} > > Nagios logs: > > - CMS Old Gen 99%(3.3GB),threadpool cache 100%(q2/c4/m4)###WARN### mem > 81% > - threadpool cache 100%(q4/c4/m4) > - CMS Old Gen 99%(3.3GB)###WARN### mem 80%,jvm\_HeapMemoryUsage > 93%(c3.9GB/u3.7GB/m3.9GB),threadpool search 92%(q0/c451/m486) > > What do you think about the logs above, especially catalina.log? > We want to upgrade from c1.xlarge(8cores, 7GB RAM) to m1.xlarge(4 cores, > 15GB RAM(8GB XMX for ES)) for each node and to increase the number of > nodes. Does it make sense(Remember the high cpu load on node 1/2 with 8 > cores)? > > Best regards > Vadim > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vadim\_Kisselmann](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@Vadim\_Kisselmann](https://discuss.elastic.co/u/Vadim_Kisselmann)\ **Post date:** [June 25, 2013, 11:54am UTC](https://discuss.elastic.co/t/memory-consumption-and-shard-allocation/12524/3 "2013-06-25T11:54:37Z")
Hi Ivan, thanks for your response. Yes, we thought about the "hot shard"-problem and it seems to be so. One index with primary shards on nodes 1/2 and only replicas on node 3 has round about 300 transactions(updates&deletes) per shard all the time. Other indices max 50 on each shard. Move replicas and primarys from node to node manually solved the "high-cpu-load"issue. Memory consumption: We will move to smaller machines(less cores) with more ram(from 8 to 15GB). Our bulk loads are made with the ElasticsearchExporter(100-doc bulks, 5 indices =17 million docs) and we have default merge settings. Cheers, Vadim Am Montag, 24. Juni 2013 22:46:59 UTC+2 schrieb Ivan Brusic: > Regarding shard allocation, it could happen that some of your shards are > hotter than others. The hashing algorithm knows nothing of data locality or > your queries. You can use the Index Status API for a few metrics about the > shards, but there is not much. Try using the Allocate API to move shards > from the busy nodes to the less nodes (and vice-versa since the shard > allocator will attempt to rebalance). If there a difference? Try changing > it back. > > As far as memory consumption goes, the logs indicate you need more memory. > 🙂 How big is your bulk load? Standard merge settings? > > Cheers, > > Ivan > > On Mon, Jun 24, 2013 at 6:21 AM, Vadim Kisselmann \<[v.kiss...@gmail.com](mailto:v.kiss...@gmail.com)\ > > > wrote: > > > Hi folks, > > > > i have a few questions about memory consumption during bulk (re-)indexing > > and shard allocation. > > > > We have a small cluster on the AWS, 3 nodes, 5 indices(5 shards & 1 > > replica each = 50 active shards) and approx. 50GB overall. > > Our setup: > > > > - ES 0.19.11 with java 7. > > - ES 4GB XMX > > - CPU: Xeon 2133MhZ, 8 cores (c1.xlarge see AWS: > > [Amazon EC2 Instance-Typen – Amazon Web Services (AWS)](http://aws.amazon.com/de/ec2/instance-types/#instance-details)) > > > > _Shard allocation:_ > > During normal operation we see that almost all the primary shards are on > > node 1 und node 2. > > Node 3 has only 2 primary shards and 14 replicas. We run many facet > > queries. Is it possible that all the queries are fired only on node 1/2 and > > only on primary shards? Because we have 90% load on these nodes. > > CPU load on node 1/2 is over 50-60% all the time, on node 3 less than > > 10%. What could be wrong here? > > > > _Memory consumption:_ > > During bulk reindexing with scan/scroll we ran into small "cluster > > overload" problems, see here from node 1: > > > > Catalina.log: > > > > [2013-06-21 17:54:23,751][INFO][monitor.jvm] [search.cloud.aws] > > [gc][ConcurrentMarkSweep][796536][41146] duration [5.7s], collections > > [1]/[6s], total [5.7s]/[2h], memory [3.4gb]-\>[3.2gb]/[3.9gb], all\_pools > > {[Code Cache] [12.1mb]-\>[12.1mb]/[48mb]}{[Par Eden Space] > > [143.1mb]-\>[16.5mb]/[532.5mb]}{[Par Survivor Space] > > [0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [3.2gb]-\>[3.2gb]/[3.3gb]}{[CMS Perm Gen] > > [37.3mb]-\>[37.3mb]/[82mb]} > > > > Nagios logs: > > > > - CMS Old Gen 99%(3.3GB),threadpool cache 100%(q2/c4/m4)###WARN### > > mem 81% > > - threadpool cache 100%(q4/c4/m4) > > - CMS Old Gen 99%(3.3GB)###WARN### mem 80%,jvm\_HeapMemoryUsage > > 93%(c3.9GB/u3.7GB/m3.9GB),threadpool search 92%(q0/c451/m486) > > > > What do you think about the logs above, especially catalina.log? > > We want to upgrade from c1.xlarge(8cores, 7GB RAM) to m1.xlarge(4 cores, > > 15GB RAM(8GB XMX for ES)) for each node and to increase the number of > > nodes. Does it make sense(Remember the high cpu load on node 1/2 with 8 > > cores)? > > > > Best regards > > Vadim > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:29am UTC](https://discuss.elastic.co/t/memory-consumption-and-shard-allocation/12524/4 "2017-07-06T02:29:34Z")