# 0.16 Release Plan **URL:** \ **Category:** Elasticsearch\ **Created:** [April 17, 2011, 6:28pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245 "2011-04-17T18:28:17Z")\ **Posts on this page:** 6\ **Page:** 2
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [May 4, 2011, 8:30am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/21 "2011-05-04T08:30:09Z")
Thanks Shay. On Wed, May 4, 2011 at 10:12 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Yes, you will need to. But, I would also check what takes that memory. Node > stats can give some info. Another place that takes memory is the internal > data structures in Lucene itself to support searching, but those are not > exposed as stats. > > On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote: > > Then I'll go for that, but I guess I will have to re-index everything if I > remove 2 shards, right? > > On Wed, May 4, 2011 at 10:03 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > If you have just a single node, and don't plan to expand the size of the > index, then a single shard is enough. As to replicas, if you have a single > node, then no replicas is fine (though even with 1 replica, that replica is > not allocated on the same node), and you can always increase it if you want. > A single shard will use less memory than 3 shards. > > -shay.banon > > On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: > > Shay, > > Maybe I'm overconfiguring the ES server, as I only need it to search > products by two fields. I used the defaults: 3 shards and 1 replica, but as > long as this data is not critical whatsoever (I could regenerate it in some > hours, and it doesn't stop my business), maybe I should just give it 1 shard > and no replicas. > > What do you think could be the minimal configuration for ES to work as per > my requirements? > > Thanks. > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > They don't take 5 minutes each, they take 5 second, the first > indicates the time it took, while the second / indicates the total time. > In any case, it seems like you might need to assign more memory for this > cluster nodes. Node stats can give you a start in trying to understand where > the memory is going. > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > Today I noticed much more activity that any other days in terms of GC: > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], > leaving [612.9mb] used, max [1gb] > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving > [842.3mb] used, max [1gb] > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], > leaving [874.7mb] used, max [1gb] > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving > [673mb] used, max [1gb] > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], > leaving [700.2mb] used, max [1gb] > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving > [718.5mb] used, max [1gb] > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], > leaving [745.1mb] used, max [1gb] > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of > 5 minutes each? It's a bit scary... > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > Hi, > > I have seen some dubious messages regarding GC that ES logs lately which > seems to indicate that the API ES uses to try and log long GCs has either > changed or got broken (it uses an internal JVM API to get it). Need to look > into it. > > -shay.banon > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > Clinton, > > Thanks a lot for your interest in my GC issues... > > Out of your list, I just added: > > bootstrap: > mlockall: 1 > > to my elasticsearch.yml config file (hope the format if fine in YML), > because I already fulfilled the other hints you were suggesting: > > ulimit -l unlimited > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > Just for your evaluation, these are some samples of my GC output: > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] > [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], > leaving [255.4mb] used, max [1.1gb] > > But FYI, I've seen sometimes times in minutes... > > Regards. > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley \<[clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) > > > wrote: > > Hi Enrique > > > I also have 0.16 working on my PROD env without issues, other than > > some GC delays... > > How long are these delays? Do they cause ES to pause? > > It's quite normal to get GC messages in the logs, eg: > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], > leaving [7.1gb] used, max [14.1gb] > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving > [7.6gb] used, max [14.1gb] > > However, if any of your Java heap is sitting in swap, it can really > affect performance, and it is worth doing some work to make sure that > you never touch the swap. > > Linux can put some data into swap even when there is plenty of free > memory. > > You may already be doing this, but just in case, here is how I would set > things up: > > 1. If you have eg 8GB of memory available on your box to run ES, I would > use 50-60% of that for ES itself, and leave the rest for kernel > filesystem caches etc > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > process permission to lock memory. > > I run ES as user 'elasticsearch' and I have this line in > '/etc/security/limits.conf': > > ``` > elasticsearch - memlock unlimited > > ``` > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > the memory > > If ES is the only service you have running on the box, you may just want > to set swapoff -a > > For more, see: > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > - > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > su'ing to a different user. To enabled that, you have to uncomment this > line in /etc/pam.d/su: > > # session required pam\_limits.so > > clint ---
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [May 4, 2011, 9:15am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/22 "2011-05-04T09:15:36Z")
These are my node stats: > Where should I focus to analyze GCs? On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro \< [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com)\> wrote: > Thanks Shay. > > On Wed, May 4, 2011 at 10:12 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Yes, you will need to. But, I would also check what takes that memory. > > Node stats can give some info. Another place that takes memory is the > > internal data structures in Lucene itself to support searching, but those > > are not exposed as stats. > > > > On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote: > > > > Then I'll go for that, but I guess I will have to re-index everything if I > > remove 2 shards, right? > > > > On Wed, May 4, 2011 at 10:03 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > wrote: > > > > If you have just a single node, and don't plan to expand the size of the > > index, then a single shard is enough. As to replicas, if you have a single > > node, then no replicas is fine (though even with 1 replica, that replica is > > not allocated on the same node), and you can always increase it if you want. > > A single shard will use less memory than 3 shards. > > > > -shay.banon > > > > On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: > > > > Shay, > > > > Maybe I'm overconfiguring the ES server, as I only need it to search > > products by two fields. I used the defaults: 3 shards and 1 replica, but as > > long as this data is not critical whatsoever (I could regenerate it in some > > hours, and it doesn't stop my business), maybe I should just give it 1 shard > > and no replicas. > > > > What do you think could be the minimal configuration for ES to work as per > > my requirements? > > > > Thanks. > > > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > wrote: > > > > They don't take 5 minutes each, they take 5 second, the first > > indicates the time it took, while the second / indicates the total time. > > In any case, it seems like you might need to assign more memory for this > > cluster nodes. Node stats can give you a start in trying to understand where > > the memory is going. > > > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > > > Today I noticed much more activity that any other days in terms of GC: > > > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] > > [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], > > leaving [612.9mb] used, max [1gb] > > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] > > [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving > > [842.3mb] used, max [1gb] > > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] > > [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], > > leaving [874.7mb] used, max [1gb] > > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] > > [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving > > [673mb] used, max [1gb] > > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] > > [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], > > leaving [700.2mb] used, max [1gb] > > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] > > [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving > > [718.5mb] used, max [1gb] > > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] > > [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], > > leaving [745.1mb] used, max [1gb] > > > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of > > 5 minutes each? It's a bit scary... > > > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon \< > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > Hi, > > > > I have seen some dubious messages regarding GC that ES logs lately > > which seems to indicate that the API ES uses to try and log long GCs has > > either changed or got broken (it uses an internal JVM API to get it). Need > > to look into it. > > > > -shay.banon > > > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > > > Clinton, > > > > Thanks a lot for your interest in my GC issues... > > > > Out of your list, I just added: > > > > bootstrap: > > mlockall: 1 > > > > to my elasticsearch.yml config file (hope the format if fine in YML), > > because I already fulfilled the other hints you were suggesting: > > > > ulimit -l unlimited > > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > > > Just for your evaluation, these are some samples of my GC output: > > > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] > > [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], > > leaving [255.4mb] used, max [1.1gb] > > > > But FYI, I've seen sometimes times in minutes... > > > > Regards. > > > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley \< > > [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)\> wrote: > > > > Hi Enrique > > > > > I also have 0.16 working on my PROD env without issues, other than > > > some GC delays... > > > > How long are these delays? Do they cause ES to pause? > > > > It's quite normal to get GC messages in the logs, eg: > > > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], > > leaving [7.1gb] used, max [14.1gb] > > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving > > [7.6gb] used, max [14.1gb] > > > > However, if any of your Java heap is sitting in swap, it can really > > affect performance, and it is worth doing some work to make sure that > > you never touch the swap. > > > > Linux can put some data into swap even when there is plenty of free > > memory. > > > > You may already be doing this, but just in case, here is how I would set > > things up: > > > > 1. If you have eg 8GB of memory available on your box to run ES, I would > > use 50-60% of that for ES itself, and leave the rest for kernel > > filesystem caches etc > > > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > > process permission to lock memory. > > > > I run ES as user 'elasticsearch' and I have this line in > > '/etc/security/limits.conf': > > > > ``` > > elasticsearch - memlock unlimited > > > > ``` > > > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > > the memory > > > > If ES is the only service you have running on the box, you may just want > > to set swapoff -a > > > > For more, see: > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > > - > > > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > > su'ing to a different user. To enabled that, you have to uncomment this > > line in /etc/pam.d/su: > > > > # session required pam\_limits.so > > > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 4, 2011, 4:38pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/23 "2011-05-04T16:38:00Z")
Its less a question of the GC, it more to see what takes memory, and if you might need to increase the memory simply because of the way you use the system. It seems like hte field cache is ~300mb, which is substantial when allocating 1gb to the ES process. On Wednesday, May 4, 2011 at 12:15 PM, Enrique Medina Montenegro wrote: > These are my node stats: > > [gist:954972 · GitHub](https://gist.github.com/954972) > > Where should I focus to analyze GCs? > > On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com) wrote: > > > Thanks Shay. > > > > On Wed, May 4, 2011 at 10:12 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > Yes, you will need to. But, I would also check what takes that memory. Node stats can give some info. Another place that takes memory is the internal data structures in Lucene itself to support searching, but those are not exposed as stats. > > > On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote: > > > > > > > Then I'll go for that, but I guess I will have to re-index everything if I remove 2 shards, right? > > > > > > > > On Wed, May 4, 2011 at 10:03 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards. > > > > > > > > > > -shay.banon > > > > > On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: > > > > > > > > > > > Shay, > > > > > > > > > > > > Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas. > > > > > > > > > > > > What do you think could be the minimal configuration for ES to work as per my requirements? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going. > > > > > > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > > > > > > > > > > > > > > Today I noticed much more activity that any other days in terms of GC: > > > > > > > > > > > > > > > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb] > > > > > > > > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb] > > > > > > > > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb] > > > > > > > > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb] > > > > > > > > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb] > > > > > > > > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb] > > > > > > > > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb] > > > > > > > > > > > > > > > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary... > > > > > > > > > > > > > > > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it. > > > > > > > > > > > > > > > > > > -shay.banon > > > > > > > > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > > > > > > > > > > > > > > > > > > Clinton, > > > > > > > > > > > > > > > > > > > > Thanks a lot for your interest in my GC issues... > > > > > > > > > > > > > > > > > > > > Out of your list, I just added: > > > > > > > > > > > > > > > > > > > > bootstrap: > > > > > > > > > > mlockall: 1 > > > > > > > > > > > > > > > > > > > > to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting: > > > > > > > > > > > > > > > > > > > > ulimit -l unlimited > > > > > > > > > > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > > > > > > > > > > > > > > > > > > > Just for your evaluation, these are some samples of my GC output: > > > > > > > > > > > > > > > > > > > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb] > > > > > > > > > > > > > > > > > > > > But FYI, I've seen sometimes times in minutes... > > > > > > > > > > > > > > > > > > > > Regards. > > > > > > > > > > > > > > > > > > > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote: > > > > > > > > > > > > > > > > > > > > > Hi Enrique > > > > > > > > > > > > > > > > > > > > > > > I also have 0.16 working on my PROD env without issues, other than > > > > > > > > > > > > some GC delays... > > > > > > > > > > > > > > > > > > > > > > How long are these delays? Do they cause ES to pause? > > > > > > > > > > > > > > > > > > > > > > It's quite normal to get GC messages in the logs, eg: > > > > > > > > > > > > > > > > > > > > > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb] > > > > > > > > > > > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb] > > > > > > > > > > > > > > > > > > > > > > However, if any of your Java heap is sitting in swap, it can really > > > > > > > > > > > affect performance, and it is worth doing some work to make sure that > > > > > > > > > > > you never touch the swap. > > > > > > > > > > > > > > > > > > > > > > Linux can put some data into swap even when there is plenty of free > > > > > > > > > > > memory. > > > > > > > > > > > > > > > > > > > > > > You may already be doing this, but just in case, here is how I would set > > > > > > > > > > > things up: > > > > > > > > > > > > > > > > > > > > > > 1. If you have eg 8GB of memory available on your box to run ES, I would > > > > > > > > > > > use 50-60% of that for ES itself, and leave the rest for kernel > > > > > > > > > > > filesystem caches etc > > > > > > > > > > > > > > > > > > > > > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > > > > > > > > > > > > > > > > > > > > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > > > > > > > > > > > process permission to lock memory. > > > > > > > > > > > > > > > > > > > > > > I run ES as user 'elasticsearch' and I have this line in > > > > > > > > > > > '/etc/security/limits.conf': > > > > > > > > > > > > > > > > > > > > > > elasticsearch - memlock unlimited > > > > > > > > > > > > > > > > > > > > > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > > > > > > > > > > > the memory > > > > > > > > > > > > > > > > > > > > > > If ES is the only service you have running on the box, you may just want > > > > > > > > > > > to set swapoff -a > > > > > > > > > > > > > > > > > > > > > > For more, see: > > > > > > > > > > > > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > > > > > > > > > > > > > > > > > > > > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > > > > > > > > > > > su'ing to a different user. To enabled that, you have to uncomment this > > > > > > > > > > > line in /etc/pam.d/su: > > > > > > > > > > > > > > > > > > > > > > # session required pam\_limits.so > > > > > > > > > > > > > > > > > > > > > > clint ---
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [May 5, 2011, 10:32am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/24 "2011-05-05T10:32:33Z")
Can I set the max\_size for field cache? Where exactly? On Wed, May 4, 2011 at 6:38 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Its less a question of the GC, it more to see what takes memory, and if > you might need to increase the memory simply because of the way you use the > system. It seems like hte field cache is ~300mb, which is substantial when > allocating 1gb to the ES process. > > On Wednesday, May 4, 2011 at 12:15 PM, Enrique Medina Montenegro wrote: > > These are my node stats: > > [gist:954972 · GitHub](https://gist.github.com/954972) > > Where should I focus to analyze GCs? > > On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro \< > [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com)\> wrote: > > Thanks Shay. > > On Wed, May 4, 2011 at 10:12 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Yes, you will need to. But, I would also check what takes that memory. Node > stats can give some info. Another place that takes memory is the internal > data structures in Lucene itself to support searching, but those are not > exposed as stats. > > On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote: > > Then I'll go for that, but I guess I will have to re-index everything if I > remove 2 shards, right? > > On Wed, May 4, 2011 at 10:03 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > If you have just a single node, and don't plan to expand the size of the > index, then a single shard is enough. As to replicas, if you have a single > node, then no replicas is fine (though even with 1 replica, that replica is > not allocated on the same node), and you can always increase it if you want. > A single shard will use less memory than 3 shards. > > -shay.banon > > On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: > > Shay, > > Maybe I'm overconfiguring the ES server, as I only need it to search > products by two fields. I used the defaults: 3 shards and 1 replica, but as > long as this data is not critical whatsoever (I could regenerate it in some > hours, and it doesn't stop my business), maybe I should just give it 1 shard > and no replicas. > > What do you think could be the minimal configuration for ES to work as per > my requirements? > > Thanks. > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > They don't take 5 minutes each, they take 5 second, the first > indicates the time it took, while the second / indicates the total time. > In any case, it seems like you might need to assign more memory for this > cluster nodes. Node stats can give you a start in trying to understand where > the memory is going. > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > Today I noticed much more activity that any other days in terms of GC: > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], > leaving [612.9mb] used, max [1gb] > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving > [842.3mb] used, max [1gb] > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], > leaving [874.7mb] used, max [1gb] > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving > [673mb] used, max [1gb] > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], > leaving [700.2mb] used, max [1gb] > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving > [718.5mb] used, max [1gb] > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], > leaving [745.1mb] used, max [1gb] > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of > 5 minutes each? It's a bit scary... > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > Hi, > > I have seen some dubious messages regarding GC that ES logs lately which > seems to indicate that the API ES uses to try and log long GCs has either > changed or got broken (it uses an internal JVM API to get it). Need to look > into it. > > -shay.banon > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > Clinton, > > Thanks a lot for your interest in my GC issues... > > Out of your list, I just added: > > bootstrap: > mlockall: 1 > > to my elasticsearch.yml config file (hope the format if fine in YML), > because I already fulfilled the other hints you were suggesting: > > ulimit -l unlimited > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > Just for your evaluation, these are some samples of my GC output: > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] > [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], > leaving [255.4mb] used, max [1.1gb] > > But FYI, I've seen sometimes times in minutes... > > Regards. > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley \<[clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) > > > wrote: > > Hi Enrique > > > I also have 0.16 working on my PROD env without issues, other than > > some GC delays... > > How long are these delays? Do they cause ES to pause? > > It's quite normal to get GC messages in the logs, eg: > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], > leaving [7.1gb] used, max [14.1gb] > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving > [7.6gb] used, max [14.1gb] > > However, if any of your Java heap is sitting in swap, it can really > affect performance, and it is worth doing some work to make sure that > you never touch the swap. > > Linux can put some data into swap even when there is plenty of free > memory. > > You may already be doing this, but just in case, here is how I would set > things up: > > 1. If you have eg 8GB of memory available on your box to run ES, I would > use 50-60% of that for ES itself, and leave the rest for kernel > filesystem caches etc > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > process permission to lock memory. > > I run ES as user 'elasticsearch' and I have this line in > '/etc/security/limits.conf': > > ``` > elasticsearch - memlock unlimited > > ``` > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > the memory > > If ES is the only service you have running on the box, you may just want > to set swapoff -a > > For more, see: > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > - > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > su'ing to a different user. To enabled that, you have to uncomment this > line in /etc/pam.d/su: > > # session required pam\_limits.so > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 5, 2011, 9:35pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/25 "2011-05-05T21:35:53Z")
Its not recommended to set it, since the field cache is a heavy one, that can potentially take time to recreate. So, if you are going to use it on a specific field, you want to keep it loaded and not unload it. Field cache is used when sorting on a field or faceting on it. On Thursday, May 5, 2011 at 1:32 PM, Enrique Medina Montenegro wrote: Can I set the max\_size for field cache? Where exactly? > On Wed, May 4, 2011 at 6:38 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Its less a question of the GC, it more to see what takes memory, and if you might need to increase the memory simply because of the way you use the system. It seems like hte field cache is ~300mb, which is substantial when allocating 1gb to the ES process. > > On Wednesday, May 4, 2011 at 12:15 PM, Enrique Medina Montenegro wrote: > > > > > These are my node stats: > > > > > > [gist:954972 · GitHub](https://gist.github.com/954972) > > > > > > Where should I focus to analyze GCs? > > > > > > On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com) wrote: > > > > > > > Thanks Shay. > > > > > > > > On Wed, May 4, 2011 at 10:12 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > Yes, you will need to. But, I would also check what takes that memory. Node stats can give some info. Another place that takes memory is the internal data structures in Lucene itself to support searching, but those are not exposed as stats. > > > > > On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote: > > > > > > > > > > > Then I'll go for that, but I guess I will have to re-index everything if I remove 2 shards, right? > > > > > > > > > > > > On Wed, May 4, 2011 at 10:03 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards. > > > > > > > > > > > > > > -shay.banon > > > > > > > On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: > > > > > > > > > > > > > > > Shay, > > > > > > > > > > > > > > > > Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas. > > > > > > > > > > > > > > > > What do you think could be the minimal configuration for ES to work as per my requirements? > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > > > > > They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going. > > > > > > > > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > > > > > > > > > > > > > > > > > > Today I noticed much more activity that any other days in terms of GC: > > > > > > > > > > > > > > > > > > > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb] > > > > > > > > > > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb] > > > > > > > > > > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb] > > > > > > > > > > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb] > > > > > > > > > > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb] > > > > > > > > > > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb] > > > > > > > > > > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb] > > > > > > > > > > > > > > > > > > > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary... > > > > > > > > > > > > > > > > > > > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it. > > > > > > > > > > > > > > > > > > > > > > -shay.banon > > > > > > > > > > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > > > > > > > > > > > > > > > > > > > > > > Clinton, > > > > > > > > > > > > > > > > > > > > > > > > Thanks a lot for your interest in my GC issues... > > > > > > > > > > > > > > > > > > > > > > > > Out of your list, I just added: > > > > > > > > > > > > > > > > > > > > > > > > bootstrap: > > > > > > > > > > > > mlockall: 1 > > > > > > > > > > > > > > > > > > > > > > > > to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting: > > > > > > > > > > > > > > > > > > > > > > > > ulimit -l unlimited > > > > > > > > > > > > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > > > > > > > > > > > > > > > > > > > > > > > Just for your evaluation, these are some samples of my GC output: > > > > > > > > > > > > > > > > > > > > > > > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb] > > > > > > > > > > > > > > > > > > > > > > > > But FYI, I've seen sometimes times in minutes... > > > > > > > > > > > > > > > > > > > > > > > > Regards. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Enrique > > > > > > > > > > > > > > > > > > > > > > > > > > > I also have 0.16 working on my PROD env without issues, other than > > > > > > > > > > > > > > some GC delays... > > > > > > > > > > > > > > > > > > > > > > > > > > How long are these delays? Do they cause ES to pause? > > > > > > > > > > > > > > > > > > > > > > > > > > It's quite normal to get GC messages in the logs, eg: > > > > > > > > > > > > > > > > > > > > > > > > > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb] > > > > > > > > > > > > > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb] > > > > > > > > > > > > > > > > > > > > > > > > > > However, if any of your Java heap is sitting in swap, it can really > > > > > > > > > > > > > affect performance, and it is worth doing some work to make sure that > > > > > > > > > > > > > you never touch the swap. > > > > > > > > > > > > > > > > > > > > > > > > > > Linux can put some data into swap even when there is plenty of free > > > > > > > > > > > > > memory. > > > > > > > > > > > > > > > > > > > > > > > > > > You may already be doing this, but just in case, here is how I would set > > > > > > > > > > > > > things up: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. If you have eg 8GB of memory available on your box to run ES, I would > > > > > > > > > > > > > use 50-60% of that for ES itself, and leave the rest for kernel > > > > > > > > > > > > > filesystem caches etc > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > > > > > > > > > > > > > process permission to lock memory. > > > > > > > > > > > > > > > > > > > > > > > > > > I run ES as user 'elasticsearch' and I have this line in > > > > > > > > > > > > > '/etc/security/limits.conf': > > > > > > > > > > > > > > > > > > > > > > > > > > elasticsearch - memlock unlimited > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > > > > > > > > > > > > > the memory > > > > > > > > > > > > > > > > > > > > > > > > > > If ES is the only service you have running on the box, you may just want > > > > > > > > > > > > > to set swapoff -a > > > > > > > > > > > > > > > > > > > > > > > > > > For more, see: > > > > > > > > > > > > > > > > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > > > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > > > > > > > > > > > > > > > > > > > > > > > > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > > > > > > > > > > > > > su'ing to a different user. To enabled that, you have to uncomment this > > > > > > > > > > > > > line in /etc/pam.d/su: > > > > > > > > > > > > > > > > > > > > > > > > > > # session required pam\_limits.so > > > > > > > > > > > > > > > > > > > > > > > > > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/26 "2017-07-06T04:06:51Z")
Closed on July 6, 2017, 4:06am UTC [Previous page](https://discuss.elastic.co/t/0-16-release-plan/4245.md?page=1)