# Elasticsearch Load Test goes OutOfMemory **URL:** \ **Category:** Elasticsearch\ **Created:** [February 25, 2015, 5:28am UTC](https://discuss.elastic.co/t/elasticsearch-load-test-goes-outofmemory/22371 "2015-02-25T05:28:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Malaka\_Gallage](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malaka_gallage/32/873_2.png) [@Malaka\_Gallage](https://discuss.elastic.co/u/Malaka_Gallage)\ **Post date:** [February 25, 2015, 5:28am UTC](https://discuss.elastic.co/t/elasticsearch-load-test-goes-outofmemory/22371/1 "2015-02-25T05:28:56Z")
Hi all, I need some help here. I started a load test for Elasticsearch before using that in production environment. I have three EC2 instances that are configured in following manner which creates a Elasticsearch cluster. All three machines has the following same hardware configurations. 32GB RAM 160GB SSD hard disk 8 core CPU _Machine 01_ Elasticsearch server (16GB heap) Elasticsearch Java client (Who generates a continues load and report to ES - 4GB heap) _Machine 02_ Elasticsearch server (16GB heap) Elasticsearch Java client (Who generates a continues load and report to ES - 4GB heap) _Machine 03_ Elasticsearch server (16GB heap) Elasticsearch Java client (Who queries from ES continuously - 1GB heap) Note that the two clients together generates around 20K records per second and report them as bulks with average size of 25. The other client queries only one query per second. My document has the following format. { "\_index": "my\_index", "\_type": "my\_type", "\_id": "7334236299916134105", "\_score": 3.6111107, "\_source": { "long\_1": 96186289301793, "long\_2": 7334236299916134000, "string\_1": "random\_string", "long\_3": 96186289301793, "string\_2": "random\_string", "string\_3": "random\_string", "string\_4": "random\_string", "string\_5": "random\_string", "long\_4": 5457314198948537000 } } The problem is, after few minutes, Elasticsearch reports errors in the logs like this. [2015-02-24 08:03:58,070][ERROR][marvel.agent.exporter] [Gateway] create failure (index:[.marvel-2015.02.24] type: [cluster\_stats]): RemoteTransportException[[Marvel Girl][inet[/10.167.199.140:9300]][bulk/shard]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@76dbf01]; [2015-02-25 04:23:36,459][ERROR][marvel.agent.exporter] [Wildside] create failure (index:[.marvel-2015.02.25] type: [index\_stats]): UnavailableShardsException[[.marvel-2015.02.25][0] [2] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@2e7693b7] Note that this error happens for different indices and different types. Again after few minutes, Elasticsearch clients get NoNodeAvailableException. I hope that is because Elasticsearch cluster malfunctioning due to above errors. But eventually the clients get "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. I did some profiling and found out that increasing the org.elasticsearch.action.index.IndexRequest instances is the cause for this OutOfMemory error. I tried even with "index.store.type: memory" and it seems still the Elasticsearch cluster cannot build the indices to the required rate. Please point out any tuning parameters or any method to get rid of these issues. Or please explain a different way to report and query this amount of load. Thanks Malaka -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 25, 2015, 6:55am UTC](https://discuss.elastic.co/t/elasticsearch-load-test-goes-outofmemory/22371/2 "2015-02-25T06:55:25Z")
If you are getting queue capacity rejections then you are over working your cluster. Are you using the bulk API for your tests? How much data is in your cluster when you get OOM? On 25 February 2015 at 16:28, Malaka Gallage [mpgallage@gmail.com](mailto:mpgallage@gmail.com) wrote: > Hi all, > > I need some help here. I started a load test for Elasticsearch before > using that in production environment. I have three EC2 instances that are > configured in following manner which creates a Elasticsearch cluster. > > All three machines has the following same hardware configurations. > > 32GB RAM > 160GB SSD hard disk > 8 core CPU > > _Machine 01_ > Elasticsearch server (16GB heap) > Elasticsearch Java client (Who generates a continues load and report to ES > > - 4GB heap) > > _Machine 02_ > Elasticsearch server (16GB heap) > Elasticsearch Java client (Who generates a continues load and report to > ES - 4GB heap) > > _Machine 03_ > Elasticsearch server (16GB heap) > Elasticsearch Java client (Who queries from ES continuously - 1GB heap) > > Note that the two clients together generates around 20K records per second > and report them as bulks with average size of 25. The other client queries > only one query per second. My document has the following format. > > { > "\_index": "my\_index", > "\_type": "my\_type", > "\_id": "7334236299916134105", > "\_score": 3.6111107, > "\_source": { > "long\_1": 96186289301793, > "long\_2": 7334236299916134000, > "string\_1": "random\_string", > "long\_3": 96186289301793, > "string\_2": "random\_string", > "string\_3": "random\_string", > "string\_4": "random\_string", > "string\_5": "random\_string", > "long\_4": 5457314198948537000 > } > } > > The problem is, after few minutes, Elasticsearch reports errors in the > logs like this. > > [2015-02-24 08:03:58,070][ERROR][marvel.agent.exporter] [Gateway] > create failure (index:[.marvel-2015.02.24] type: [cluster\_stats]): > RemoteTransportException[[Marvel Girl][inet[/10.167.199.140:9300]][bulk/shard]]; > nested: EsRejectedExecutionException[rejected execution (queue capacity 50) > on > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@76dbf01 > ]; > > [2015-02-25 04:23:36,459][ERROR][marvel.agent.exporter] [Wildside] > create failure (index:[.marvel-2015.02.25] type: [index\_stats]): > UnavailableShardsException[[.marvel-2015.02.25][0] [2] shardIt, [0] active > : Timeout waiting for [1m], request: > org.elasticsearch.action.bulk.BulkShardRequest@2e7693b7] > > Note that this error happens for different indices and different types. > > Again after few minutes, Elasticsearch clients get > NoNodeAvailableException. I hope that is because Elasticsearch cluster > malfunctioning due to above errors. But eventually the clients get > "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. > > I did some profiling and found out that increasing > the org.elasticsearch.action.index.IndexRequest instances is the cause for > this OutOfMemory error. I tried even with "index.store.type: memory" and it > seems still the Elasticsearch cluster cannot build the indices to the > required rate. > > Please point out any tuning parameters or any method to get rid of these > issues. Or please explain a different way to report and query this amount > of load. > > Thanks > Malaka > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8xOtuTTo4h5WfN7Q2G87yxt3KoODHOXbJ1F6PfOd7CNg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8xOtuTTo4h5WfN7Q2G87yxt3KoODHOXbJ1F6PfOd7CNg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Malaka\_Gallage](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malaka_gallage/32/873_2.png) [@Malaka\_Gallage](https://discuss.elastic.co/u/Malaka_Gallage)\ **Post date:** [February 25, 2015, 11:08am UTC](https://discuss.elastic.co/t/elasticsearch-load-test-goes-outofmemory/22371/3 "2015-02-25T11:08:47Z")
Hi Mark, Yes I'm using bulk API for the tests. Usually OOM error happens when the cluster has around 30 million records. Is there anyway to tune the ES cluster to perform better? Thanks Malaka On Wednesday, February 25, 2015 at 12:25:52 PM UTC+5:30, Mark Walkom wrote: > If you are getting queue capacity rejections then you are over working > your cluster. Are you using the bulk API for your tests? > How much data is in your cluster when you get OOM? > > On 25 February 2015 at 16:28, Malaka Gallage \<[mpga...@gmail.com](mailto:mpga...@gmail.com) > \\> wrote: > > > Hi all, > > > > I need some help here. I started a load test for Elasticsearch before > > using that in production environment. I have three EC2 instances that are > > configured in following manner which creates a Elasticsearch cluster. > > > > All three machines has the following same hardware configurations. > > > > 32GB RAM > > 160GB SSD hard disk > > 8 core CPU > > > > _Machine 01_ > > Elasticsearch server (16GB heap) > > Elasticsearch Java client (Who generates a continues load and report to > > ES - 4GB heap) > > > > _Machine 02_ > > Elasticsearch server (16GB heap) > > Elasticsearch Java client (Who generates a continues load and report to > > ES - 4GB heap) > > > > _Machine 03_ > > Elasticsearch server (16GB heap) > > Elasticsearch Java client (Who queries from ES continuously - 1GB heap) > > > > Note that the two clients together generates around 20K records per > > second and report them as bulks with average size of 25. The other client > > queries only one query per second. My document has the following format. > > > > { > > "\_index": "my\_index", > > "\_type": "my\_type", > > "\_id": "7334236299916134105", > > "\_score": 3.6111107, > > "\_source": { > > "long\_1": 96186289301793, > > "long\_2": 7334236299916134000, > > "string\_1": "random\_string", > > "long\_3": 96186289301793, > > "string\_2": "random\_string", > > "string\_3": "random\_string", > > "string\_4": "random\_string", > > "string\_5": "random\_string", > > "long\_4": 5457314198948537000 > > } > > } > > > > The problem is, after few minutes, Elasticsearch reports errors in the > > logs like this. > > > > [2015-02-24 08:03:58,070][ERROR][marvel.agent.exporter] [Gateway] > > create failure (index:[.marvel-2015.02.24] type: [cluster\_stats]): > > RemoteTransportException[[Marvel > > Girl][inet[/10.167.199.140:9300]][bulk/shard]]; nested: > > EsRejectedExecutionException[rejected execution (queue capacity 50) on > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@76dbf01]; > > > > [2015-02-25 04:23:36,459][ERROR][marvel.agent.exporter] [Wildside] > > create failure (index:[.marvel-2015.02.25] type: [index\_stats]): > > UnavailableShardsException[[.marvel-2015.02.25][0] [2] shardIt, [0] active > > : Timeout waiting for [1m], request: > > org.elasticsearch.action.bulk.BulkShardRequest@2e7693b7] > > > > Note that this error happens for different indices and different types. > > > > Again after few minutes, Elasticsearch clients get > > NoNodeAvailableException. I hope that is because Elasticsearch cluster > > malfunctioning due to above errors. But eventually the clients get > > "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. > > > > I did some profiling and found out that increasing > > the org.elasticsearch.action.index.IndexRequest instances is the cause for > > this OutOfMemory error. I tried even with "index.store.type: memory" and it > > seems still the Elasticsearch cluster cannot build the indices to the > > required rate. > > > > Please point out any tuning parameters or any method to get rid of these > > issues. Or please explain a different way to report and query this amount > > of load. > > > > Thanks > > Malaka > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 26, 2015, 12:56am UTC](https://discuss.elastic.co/t/elasticsearch-load-test-goes-outofmemory/22371/4 "2015-02-26T00:56:15Z")
What sort of queries are you running? On 25 February 2015 at 22:08, Malaka Gallage [mpgallage@gmail.com](mailto:mpgallage@gmail.com) wrote: > Hi Mark, > > Yes I'm using bulk API for the tests. Usually OOM error happens when the > cluster has around 30 million records. Is there anyway to tune the ES > cluster to perform better? > > Thanks > Malaka > > On Wednesday, February 25, 2015 at 12:25:52 PM UTC+5:30, Mark Walkom wrote: > > > If you are getting queue capacity rejections then you are over working > > your cluster. Are you using the bulk API for your tests? > > How much data is in your cluster when you get OOM? > > > > On 25 February 2015 at 16:28, Malaka Gallage [mpga...@gmail.com](mailto:mpga...@gmail.com) wrote: > > > > > Hi all, > > > > > > I need some help here. I started a load test for Elasticsearch before > > > using that in production environment. I have three EC2 instances that are > > > configured in following manner which creates a Elasticsearch cluster. > > > > > > All three machines has the following same hardware configurations. > > > > > > 32GB RAM > > > 160GB SSD hard disk > > > 8 core CPU > > > > > > _Machine 01_ > > > Elasticsearch server (16GB heap) > > > Elasticsearch Java client (Who generates a continues load and report to > > > ES - 4GB heap) > > > > > > _Machine 02_ > > > Elasticsearch server (16GB heap) > > > Elasticsearch Java client (Who generates a continues load and report to > > > ES - 4GB heap) > > > > > > _Machine 03_ > > > Elasticsearch server (16GB heap) > > > Elasticsearch Java client (Who queries from ES continuously - 1GB heap) > > > > > > Note that the two clients together generates around 20K records per > > > second and report them as bulks with average size of 25. The other client > > > queries only one query per second. My document has the following format. > > > > > > { > > > "\_index": "my\_index", > > > "\_type": "my\_type", > > > "\_id": "7334236299916134105", > > > "\_score": 3.6111107, > > > "\_source": { > > > "long\_1": 96186289301793, > > > "long\_2": 7334236299916134000, > > > "string\_1": "random\_string", > > > "long\_3": 96186289301793, > > > "string\_2": "random\_string", > > > "string\_3": "random\_string", > > > "string\_4": "random\_string", > > > "string\_5": "random\_string", > > > "long\_4": 5457314198948537000 > > > } > > > } > > > > > > The problem is, after few minutes, Elasticsearch reports errors in the > > > logs like this. > > > > > > [2015-02-24 08:03:58,070][ERROR][marvel.agent.exporter] [Gateway] > > > create failure (index:[.marvel-2015.02.24] type: [cluster\_stats]): > > > RemoteTransportException[[Marvel Girl][inet[/10.167.199.140:9300]][bulk/shard]]; > > > nested: EsRejectedExecutionException[rejected execution (queue capacity > > > 50) on org.elasticsearch.action.support.replication. > > > TransportShardReplicationOperationAction$AsyncShardOperationAction$1@ > > > 76dbf01]; > > > > > > [2015-02-25 04:23:36,459][ERROR][marvel.agent.exporter] [Wildside] > > > create failure (index:[.marvel-2015.02.25] type: [index\_stats]): > > > UnavailableShardsException[[.marvel-2015.02.25][0] [2] shardIt, [0] > > > active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk. > > > BulkShardRequest@2e7693b7] > > > > > > Note that this error happens for different indices and different types. > > > > > > Again after few minutes, Elasticsearch clients get > > > NoNodeAvailableException. I hope that is because Elasticsearch cluster > > > malfunctioning due to above errors. But eventually the clients get > > > "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. > > > > > > I did some profiling and found out that increasing > > > the org.elasticsearch.action.index.IndexRequest instances is the cause > > > for this OutOfMemory error. I tried even with "index.store.type: memory" > > > and it seems still the Elasticsearch cluster cannot build the indices to > > > the required rate. > > > > > > Please point out any tuning parameters or any method to get rid of these > > > issues. Or please explain a different way to report and query this amount > > > of load. > > > > > > Thanks > > > Malaka > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9BsKyGa-CNEZA8VZG05vdLCts0pvSawr0LFiMWYz57TQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9BsKyGa-CNEZA8VZG05vdLCts0pvSawr0LFiMWYz57TQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Malaka\_Gallage](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malaka_gallage/32/873_2.png) [@Malaka\_Gallage](https://discuss.elastic.co/u/Malaka_Gallage)\ **Post date:** [February 26, 2015, 4:23am UTC](https://discuss.elastic.co/t/elasticsearch-load-test-goes-outofmemory/22371/5 "2015-02-26T04:23:48Z")
Hi Mark, I run one random query out of four defined queries at a time. 1. Getting the average of a field over some time period. 2. Getting the max of a field over some time period. 3. Getting the min of a field over some time period. 4. Getting a percentile of a field over some time period. Note that one of this query runs only once a second. Thanks Malaka On Thursday, February 26, 2015 at 6:26:51 AM UTC+5:30, Mark Walkom wrote: > What sort of queries are you running? > > On 25 February 2015 at 22:08, Malaka Gallage \<[mpga...@gmail.com](mailto:mpga...@gmail.com) > \\> wrote: > > > Hi Mark, > > > > Yes I'm using bulk API for the tests. Usually OOM error happens when the > > cluster has around 30 million records. Is there anyway to tune the ES > > cluster to perform better? > > > > Thanks > > Malaka > > > > On Wednesday, February 25, 2015 at 12:25:52 PM UTC+5:30, Mark Walkom > > wrote: > > > > > If you are getting queue capacity rejections then you are over working > > > your cluster. Are you using the bulk API for your tests? > > > How much data is in your cluster when you get OOM? > > > > > > On 25 February 2015 at 16:28, Malaka Gallage [mpga...@gmail.com](mailto:mpga...@gmail.com) wrote: > > > > > > > Hi all, > > > > > > > > I need some help here. I started a load test for Elasticsearch before > > > > using that in production environment. I have three EC2 instances that are > > > > configured in following manner which creates a Elasticsearch cluster. > > > > > > > > All three machines has the following same hardware configurations. > > > > > > > > 32GB RAM > > > > 160GB SSD hard disk > > > > 8 core CPU > > > > > > > > _Machine 01_ > > > > Elasticsearch server (16GB heap) > > > > Elasticsearch Java client (Who generates a continues load and report to > > > > ES - 4GB heap) > > > > > > > > _Machine 02_ > > > > Elasticsearch server (16GB heap) > > > > Elasticsearch Java client (Who generates a continues load and report to > > > > ES - 4GB heap) > > > > > > > > _Machine 03_ > > > > Elasticsearch server (16GB heap) > > > > Elasticsearch Java client (Who queries from ES continuously - 1GB heap) > > > > > > > > Note that the two clients together generates around 20K records per > > > > second and report them as bulks with average size of 25. The other client > > > > queries only one query per second. My document has the following format. > > > > > > > > { > > > > "\_index": "my\_index", > > > > "\_type": "my\_type", > > > > "\_id": "7334236299916134105", > > > > "\_score": 3.6111107, > > > > "\_source": { > > > > "long\_1": 96186289301793, > > > > "long\_2": 7334236299916134000, > > > > "string\_1": "random\_string", > > > > "long\_3": 96186289301793, > > > > "string\_2": "random\_string", > > > > "string\_3": "random\_string", > > > > "string\_4": "random\_string", > > > > "string\_5": "random\_string", > > > > "long\_4": 5457314198948537000 > > > > } > > > > } > > > > > > > > The problem is, after few minutes, Elasticsearch reports errors in the > > > > logs like this. > > > > > > > > [2015-02-24 08:03:58,070][ERROR][marvel.agent.exporter] [Gateway] > > > > create failure (index:[.marvel-2015.02.24] type: [cluster\_stats]): > > > > RemoteTransportException[[Marvel Girl][inet[/10.167.199.140:9300]][bulk/shard]]; > > > > nested: EsRejectedExecutionException[rejected execution (queue > > > > capacity 50) on org.elasticsearch.action.support.replication. > > > > TransportShardReplicationOperationAction$AsyncShardOperationAction$1@ > > > > 76dbf01]; > > > > > > > > [2015-02-25 04:23:36,459][ERROR][marvel.agent.exporter] [Wildside] > > > > create failure (index:[.marvel-2015.02.25] type: [index\_stats]): > > > > UnavailableShardsException[[.marvel-2015.02.25][0] [2] shardIt, [0] > > > > active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk. > > > > BulkShardRequest@2e7693b7] > > > > > > > > Note that this error happens for different indices and different types. > > > > > > > > Again after few minutes, Elasticsearch clients get > > > > NoNodeAvailableException. I hope that is because Elasticsearch cluster > > > > malfunctioning due to above errors. But eventually the clients get > > > > "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. > > > > > > > > I did some profiling and found out that increasing > > > > the org.elasticsearch.action.index.IndexRequest instances is the cause > > > > for this OutOfMemory error. I tried even with "index.store.type: memory" > > > > and it seems still the Elasticsearch cluster cannot build the indices to > > > > the required rate. > > > > > > > > Please point out any tuning parameters or any method to get rid of > > > > these issues. Or please explain a different way to report and query this > > > > amount of load. > > > > > > > > Thanks > > > > Malaka > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/35a29ca5-02f6-4fe9-8600-2cdb91c519cf%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a78a83f3-090d-4265-8d8c-1fe0d669e70b%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/48458521-a59b-4c16-9bbf-e4b21cb94d43%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/48458521-a59b-4c16-9bbf-e4b21cb94d43%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:29am UTC](https://discuss.elastic.co/t/elasticsearch-load-test-goes-outofmemory/22371/6 "2017-07-06T00:29:59Z")