# 0.16 Release Plan **URL:** \ **Category:** Elasticsearch\ **Created:** [April 17, 2011, 6:28pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245 "2011-04-17T18:28:17Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 17, 2011, 6:28pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/1 "2011-04-17T18:28:17Z")
Heya, 0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin ([http://www.elasticsearch.org/download/master/](http://www.elasticsearch.org/download/master/)). For the Java users, snapshot version have been uploaded as well. -shay.banon ---
**Author:** ![K\_B](https://avatars.discourse-cdn.com/v4/letter/k/278dde/32.png) [@K\_B](https://discuss.elastic.co/u/K_B)\ **Post date:** [April 17, 2011, 6:45pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/2 "2011-04-17T18:45:34Z")
Heya Shay, just wanted to say thank you for the much work you've done on that. Really appreciate it even if we had differend pov's in the past. Keep on! Best, KB On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Heya, > > 0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > For the Java users, snapshot version have been uploaded as well. > > -shay.banon ---
**Author:** ![ruflin\_2](https://avatars.discourse-cdn.com/v4/letter/r/a8b319/32.png) [@ruflin\_2](https://discuss.elastic.co/u/ruflin_2)\ **Post date:** [April 17, 2011, 8:41pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/3 "2011-04-17T20:41:14Z")
Hi Shay I used 0.16 in the last days. I did not find any issues. 0.16 passes all Elastica tests 😉 Thank you for all your work. I'm really looking forward to 0.16, especially Tika 0.9 because I was having some troubles with the last version. Nicolas On Apr 17, 8:45 pm, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote: > Heya Shay, > > just wanted to say thank you for the much work you've done on that. > Really appreciate it even if we had differend pov's in the past. > > Keep on! > > Best, > > KB > > On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Heya, > > > 0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > > For the Java users, snapshot version have been uploaded as well. > > > -shay.banon ---
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [April 18, 2011, 9:09am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/4 "2011-04-18T09:09:25Z")
Shay, I also have 0.16 working on my PROD env without issues, other than some GC delays... Thanks for the great work! On Sun, Apr 17, 2011 at 10:41 PM, ruflin [ruflin@gmail.com](mailto:ruflin@gmail.com) wrote: > Hi Shay > > I used 0.16 in the last days. I did not find any issues. 0.16 passes > all Elastica tests 😉 > > Thank you for all your work. I'm really looking forward to 0.16, > especially Tika 0.9 because I was having some troubles with the last > version. > > Nicolas > > On Apr 17, 8:45 pm, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote: > > > Heya Shay, > > > > just wanted to say thank you for the much work you've done on that. > > Really appreciate it even if we had differend pov's in the past. > > > > Keep on! > > > > Best, > > > > KB > > > > On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Heya, > > > > > 0.16 is just around the corner. Basically, master is pretty much > > > feature complete, so, if you have time, take it for a spin ( > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > > > > For the Java users, snapshot version have been uploaded as well. > > > > > -shay.banon ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [April 18, 2011, 9:28am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/5 "2011-04-18T09:28:19Z")
Hi Enrique > I also have 0.16 working on my PROD env without issues, other than > some GC delays... How long are these delays? Do they cause ES to pause? It's quite normal to get GC messages in the logs, eg: [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb] [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb] However, if any of your Java heap is sitting in swap, it can really affect performance, and it is worth doing some work to make sure that you never touch the swap. Linux can put some data into swap even when there is plenty of free memory. You may already be doing this, but just in case, here is how I would set things up: 1. If you have eg 8GB of memory available on your box to run ES, I would use 50-60% of that for ES itself, and leave the rest for kernel filesystem caches etc 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) 3. Set "ulimit -l unlimited" before starting ES - this gives the ES process permission to lock memory. 4. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock the memory If ES is the only service you have running on the box, you may just want to set swapoff -a For more, see: - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) Note: Ubuntu doesn't read /etc/security/limits.conf by default, when su'ing to a different user. To enabled that, you have to uncomment this line in /etc/pam.d/su: ``` # session required pam_limits.so ``` clint ---
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [April 18, 2011, 10:00am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/6 "2011-04-18T10:00:38Z")
Clinton, Thanks a lot for your interest in my GC issues... Out of your list, I just added: bootstrap: mlockall: 1 to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting: ulimit -l unlimited /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & Just for your evaluation, these are some samples of my GC output: [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb] But FYI, I've seen sometimes times in minutes... Regards. On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > Hi Enrique > > > I also have 0.16 working on my PROD env without issues, other than > > some GC delays... > > How long are these delays? Do they cause ES to pause? > > It's quite normal to get GC messages in the logs, eg: > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], > leaving [7.1gb] used, max [14.1gb] > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving > [7.6gb] used, max [14.1gb] > > However, if any of your Java heap is sitting in swap, it can really > affect performance, and it is worth doing some work to make sure that > you never touch the swap. > > Linux can put some data into swap even when there is plenty of free > memory. > > You may already be doing this, but just in case, here is how I would set > things up: > > 1. If you have eg 8GB of memory available on your box to run ES, I would > use 50-60% of that for ES itself, and leave the rest for kernel > filesystem caches etc > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > process permission to lock memory. > > I run ES as user 'elasticsearch' and I have this line in > '/etc/security/limits.conf': > > ``` > elasticsearch - memlock unlimited > > ``` > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > the memory > > If ES is the only service you have running on the box, you may just want > to set swapoff -a > > For more, see: > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > - > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > su'ing to a different user. To enabled that, you have to uncomment this > line in /etc/pam.d/su: > > # session required pam\_limits.so > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 19, 2011, 9:04am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/7 "2011-04-19T09:04:55Z")
Hi, I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it. -shay.banon On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > Clinton, > > Thanks a lot for your interest in my GC issues... > > Out of your list, I just added: > > bootstrap: > mlockall: 1 > > to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting: > > ulimit -l unlimited > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > Just for your evaluation, these are some samples of my GC output: > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb] > > But FYI, I've seen sometimes times in minutes... > > Regards. > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote: > > > Hi Enrique > > > > > I also have 0.16 working on my PROD env without issues, other than > > > some GC delays... > > > > How long are these delays? Do they cause ES to pause? > > > > It's quite normal to get GC messages in the logs, eg: > > > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb] > > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb] > > > > However, if any of your Java heap is sitting in swap, it can really > > affect performance, and it is worth doing some work to make sure that > > you never touch the swap. > > > > Linux can put some data into swap even when there is plenty of free > > memory. > > > > You may already be doing this, but just in case, here is how I would set > > things up: > > > > 1. If you have eg 8GB of memory available on your box to run ES, I would > > use 50-60% of that for ES itself, and leave the rest for kernel > > filesystem caches etc > > > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > > process permission to lock memory. > > > > I run ES as user 'elasticsearch' and I have this line in > > '/etc/security/limits.conf': > > > > elasticsearch - memlock unlimited > > > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > > the memory > > > > If ES is the only service you have running on the box, you may just want > > to set swapoff -a > > > > For more, see: > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > > su'ing to a different user. To enabled that, you have to uncomment this > > line in /etc/pam.d/su: > > > > # session required pam\_limits.so > > > > clint ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [April 19, 2011, 9:11am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/8 "2011-04-19T09:11:08Z")
Hi, Hate to tell this but I encountered some issues. I upgraded to the recent master and found out that I got hanging thread during indexing (I do not have a rigorous test for this, I just run indexing of data manually and 3x in row I got some hanging thread, this can be coincidence but if it is then I would say quite rare). On the other hand when I downgraded to commit > then all my data was indexed without any problems. To me it seems that between current master and the said commit there can be some problem, at least that is what it looks like on my side. Regards, Lukas On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro \< [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com)\> wrote: > Shay, > > I also have 0.16 working on my PROD env without issues, other than some GC > delays... > > Thanks for the great work! > > On Sun, Apr 17, 2011 at 10:41 PM, ruflin [ruflin@gmail.com](mailto:ruflin@gmail.com) wrote: > > > Hi Shay > > > > I used 0.16 in the last days. I did not find any issues. 0.16 passes > > all Elastica tests 😉 > > > > Thank you for all your work. I'm really looking forward to 0.16, > > especially Tika 0.9 because I was having some troubles with the last > > version. > > > > Nicolas > > > > On Apr 17, 8:45 pm, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote: > > > > > Heya Shay, > > > > > > just wanted to say thank you for the much work you've done on that. > > > Really appreciate it even if we had differend pov's in the past. > > > > > > Keep on! > > > > > > Best, > > > > > > KB > > > > > > On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Heya, > > > > > > > 0.16 is just around the corner. Basically, master is pretty much > > > > feature complete, so, if you have time, take it for a spin ( > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > > > > > > For the Java users, snapshot version have been uploaded as well. > > > > > > > -shay.banon ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 19, 2011, 9:15am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/9 "2011-04-19T09:15:44Z")
Recreation? On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote: > Hi, > > Hate to tell this but I encountered some issues. I upgraded to the recent master and found out that I got hanging thread during indexing (I do not have a rigorous test for this, I just run indexing of data manually and 3x in row I got some hanging thread, this can be coincidence but if it is then I would say quite rare). On the other hand when I downgraded to commit [small refactor · elastic/elasticsearch@2cc4a28 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/2cc4a286f8ac043be4c3cf6afce2c7cc6790a4b1) then all my data was indexed without any problems. > > To me it seems that between current master and the said commit there can be some problem, at least that is what it looks like on my side. > > Regards, > Lukas > > On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com) wrote: > > > Shay, > > > > I also have 0.16 working on my PROD env without issues, other than some GC delays... > > > > Thanks for the great work! > > > > On Sun, Apr 17, 2011 at 10:41 PM, ruflin [ruflin@gmail.com](mailto:ruflin@gmail.com) wrote: > > > > > Hi Shay > > > > > > I used 0.16 in the last days. I did not find any issues. 0.16 passes > > > all Elastica tests 😉 > > > > > > Thank you for all your work. I'm really looking forward to 0.16, > > > especially Tika 0.9 because I was having some troubles with the last > > > version. > > > > > > Nicolas > > > > > > On Apr 17, 8:45 pm, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote: > > > > > > > Heya Shay, > > > > > > > > just wanted to say thank you for the much work you've done on that. > > > > Really appreciate it even if we had differend pov's in the past. > > > > > > > > Keep on! > > > > > > > > Best, > > > > > > > > KB > > > > > > > > On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > Heya, > > > > > > > > > 0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > > > > > > > > For the Java users, snapshot version have been uploaded as well. > > > > > > > > > -shay.banon ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [April 19, 2011, 9:19am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/10 "2011-04-19T09:19:34Z")
Yea I know.... probably not easy for me now... at least I will try to identify the commit that "breaks" it for me. On Tue, Apr 19, 2011 at 11:15 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Recreation? > > On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote: > > Hi, > > Hate to tell this but I encountered some issues. I upgraded to the recent > master and found out that I got hanging thread during indexing (I do not > have a rigorous test for this, I just run indexing of data manually and 3x > in row I got some hanging thread, this can be coincidence but if it is then > I would say quite rare). On the other hand when I downgraded to commit > [small refactor · elastic/elasticsearch@2cc4a28 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/2cc4a286f8ac043be4c3cf6afce2c7cc6790a4b1) then > all my data was indexed without any problems. > > To me it seems that between current master and the said commit there can be > some problem, at least that is what it looks like on my side. > > Regards, > Lukas > > On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro \< > [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com)\> wrote: > > Shay, > > I also have 0.16 working on my PROD env without issues, other than some GC > delays... > > Thanks for the great work! > > On Sun, Apr 17, 2011 at 10:41 PM, ruflin [ruflin@gmail.com](mailto:ruflin@gmail.com) wrote: > > Hi Shay > > I used 0.16 in the last days. I did not find any issues. 0.16 passes > all Elastica tests 😉 > > Thank you for all your work. I'm really looking forward to 0.16, > especially Tika 0.9 because I was having some troubles with the last > version. > > Nicolas > > On Apr 17, 8:45 pm, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote: > > > Heya Shay, > > > > just wanted to say thank you for the much work you've done on that. > > Really appreciate it even if we had differend pov's in the past. > > > > Keep on! > > > > Best, > > > > KB > > > > On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Heya, > > > > > 0.16 is just around the corner. Basically, master is pretty much > > > feature complete, so, if you have time, take it for a spin ( > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > > > > For the Java users, snapshot version have been uploaded as well. > > > > > -shay.banon ---
**Author:** ![Barsk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/barsk/32/13734_2.png) [@Barsk](https://discuss.elastic.co/u/Barsk)\ **Post date:** [April 21, 2011, 10:34am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/11 "2011-04-21T10:34:08Z")
``` Any progress on the release? I am about to set up my production environment now and I was hoping to jump on the final 0.16 version. But if there are still problems I can wait and use the 0.15.2 version. Is there any substantial differences in performance or memory usage between 0.15.2 and 0.16 that may be of interest or is it just functional upgrades (which I do not need right now)? Main question, jump on 0.16 master and build, or stick with 0.15.2? /Kristian Lukáš Vlček skrev 2011-04-19 11:19:
Yea I know.... probably not easy for me now... at least I will try to identify the commit that "breaks" it for me. ``` On Tue, Apr 19, 2011 at 11:15 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > Recreation? > > On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote: > > > Hi, > > > > Hate to tell this but I encountered some > > issues. I upgraded to the recent master and > > found out that I got hanging thread during > > indexing (I do not have a rigorous test for > > this, I just run indexing of data manually > > and 3x in row I got some hanging thread, > > this can be coincidence but if it is then I > > would say quite rare). On the other hand > > when I downgraded to commit [https://github.com/elasticsearch/elasticsearch/commit/2cc4a286f8ac043be4c3cf6afce2c7cc6790a4b1](https://github.com/elasticsearch/elasticsearch/commit/2cc4a286f8ac043be4c3cf6afce2c7cc6790a4b1) then > > all my data was indexed without any > > problems. > > > > To me it seems that between current > > master and the said commit there can be some > > problem, at least that is what it looks like > > on my side. > > > > Regards,Lukas > > > > ``` > > On Mon, Apr 18, 2011 at 11:09 AM, Enrique > > Medina Montenegro <e.medina.m@gmail.com> > > wrote: > > > > ``` > > > > > Shay, > > > > > > I also have 0.16 working on my > > > PROD env without issues, other than > > > some GC delays... > > > > > > ``` > > > Thanks for the great work! > > > > > > > > > On Sun, Apr 17, 2011 at 10:41 > > > PM, ruflin <ruflin@gmail.com> > > > wrote: > > > > > > ``` > > > > > > > Hi Shay > > > > > > > > ``` > > > > I used 0.16 in the last > > > > days. I did not find any > > > > issues. 0.16 passes > > > > > > > > all Elastica tests ;-) > > > > > > > > Thank you for all your > > > > work. I'm really looking > > > > forward to 0.16, > > > > > > > > especially Tika 0.9 > > > > because I was having some > > > > troubles with the last > > > > > > > > version. > > > > > > > > Nicolas > > > > > > > > On Apr 17, 8:45 pm, > > > > "K.B." <korbinian.ba...@googlemail.com> > > > > wrote: > > > > > > > > > Heya Shay, > > > > > > > > > > > > > > > > > > just wanted to > > > > say thank you for the > > > > much work you've done > > > > on that. > > > > > > > > > Really appreciate > > > > it even if we had > > > > differend pov's in the > > > > past. > > > > > > > > > > > > > > > > > > Keep on! > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > KB > > > > > > > > > > > > > > > > > > On 17 Apr., > > > > 20:28, Shay Banon <shay.ba...@elasticsearch.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Heya, > > > > > > > > > > > > > > > > > > > 0.16 is just > > > > around the corner. > > > > Basically, master is > > > > pretty much feature > > > > complete, so, if you > > > > have time, take it for > > > > a spin (http://www.elasticsearch.org/download/master/). > > > > > > > > > > > > > > > > > > > For the Java > > > > users, snapshot > > > > version have been > > > > uploaded as well. > > > > > > > > > > > > > > > > > > > -shay.banon > > > > > > > > ``` ``` -- Med vänlig hälsning Kristian Jörg Devo IT AB Tel: 054 - 22 14 58, 0709 - 15 83 42 E-post:[kristian.jorg@devo.se](mailto:kristian.jorg@devo.se)Webb:[http://www.devo.se](http://www.devo.se) ``` ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 21, 2011, 10:40am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/12 "2011-04-21T10:40:03Z")
Current plan is to release 0.16 either today or tomorrow (still checking if a couple of small features will make it in, and some small bug fixes open). There are a lot of improvements in 0.16, but, don't push it to prod before you tested it with your app. -shay.banon On Thursday, April 21, 2011 at 1:34 PM, Kristian Jörg wrote: > Any progress on the release? > I am about to set up my production environment now and I was hoping to jump on the final 0.16 version. But if there are still problems I can wait and use the 0.15.2 version. Is there any substantial differences in performance or memory usage between 0.15.2 and 0.16 that may be of interest or is it just functional upgrades (which I do not need right now)? > > Main question, jump on 0.16 master and build, or stick with 0.15.2? > > /Kristian > > Lukáš Vlček skrev 2011-04-19 11:19: > > > Yea I know.... probably not easy for me now... at least I will try to identify the commit that "breaks" it for me. > > > > On Tue, Apr 19, 2011 at 11:15 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > Recreation? > > > On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote: > > > > > > > Hi, > > > > > > > > Hate to tell this but I encountered some issues. I upgraded to the recent master and found out that I got hanging thread during indexing (I do not have a rigorous test for this, I just run indexing of data manually and 3x in row I got some hanging thread, this can be coincidence but if it is then I would say quite rare). On the other hand when I downgraded to commit [small refactor · elastic/elasticsearch@2cc4a28 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/2cc4a286f8ac043be4c3cf6afce2c7cc6790a4b1) then all my data was indexed without any problems. > > > > > > > > To me it seems that between current master and the said commit there can be some problem, at least that is what it looks like on my side. > > > > > > > > Regards, > > > > Lukas > > > > > > > > On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com) wrote: > > > > > > > > > Shay, > > > > > > > > > > I also have 0.16 working on my PROD env without issues, other than some GC delays... > > > > > > > > > > Thanks for the great work! > > > > > > > > > > On Sun, Apr 17, 2011 at 10:41 PM, ruflin [ruflin@gmail.com](mailto:ruflin@gmail.com) wrote: > > > > > > > > > > > Hi Shay > > > > > > > > > > > > I used 0.16 in the last days. I did not find any issues. 0.16 passes > > > > > > all Elastica tests 😉 > > > > > > > > > > > > Thank you for all your work. I'm really looking forward to 0.16, > > > > > > especially Tika 0.9 because I was having some troubles with the last > > > > > > version. > > > > > > > > > > > > Nicolas > > > > > > > > > > > > On Apr 17, 8:45 pm, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote: > > > > > > > > > > > > > Heya Shay, > > > > > > > > > > > > > > just wanted to say thank you for the much work you've done on that. > > > > > > > Really appreciate it even if we had differend pov's in the past. > > > > > > > > > > > > > > Keep on! > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > KB > > > > > > > > > > > > > > On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > > > > > > > Heya, > > > > > > > > > > > > > > > 0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > > > > > > > > > > > > > > For the Java users, snapshot version have been uploaded as well. > > > > > > > > > > > > > > > -shay.banon > > -- Med vänlig hälsning Kristian Jörg Devo IT AB Tel: 054 - 22 14 58, 0709 - 15 83 42 E-post: kristian.jorg@devo.se Webb: [http://www.devo.se](http://www.devo.se) ---
**Author:** ![Barsk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/barsk/32/13734_2.png) [@Barsk](https://discuss.elastic.co/u/Barsk)\ **Post date:** [April 21, 2011, 1:05pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/13 "2011-04-21T13:05:27Z")
Shay Banon skrev 2011-04-21 12:40: > Current plan is to release 0.16 either today or tomorrow (still > checking if a couple of small features will make it in, and some small > bug fixes open). Thanks for the update. Yes, I will start prod on 0.15.2 and test 0.16 befor upgrading, I came to that conclusion myself after a cup of coffee. 🙂 Regards Kristian ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [April 26, 2011, 12:41pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/14 "2011-04-26T12:41:10Z")
Hi, just follow up on my "hanging thread" issue, the problem was on my side. I did not update all the elaticsearch files and sources correctly on the server. It runs fine for me now. Thanks, Lukas On Tue, Apr 19, 2011 at 11:11 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Hi, > > Hate to tell this but I encountered some issues. I upgraded to the recent > master and found out that I got hanging thread during indexing (I do not > have a rigorous test for this, I just run indexing of data manually and 3x > in row I got some hanging thread, this can be coincidence but if it is then > I would say quite rare). On the other hand when I downgraded to commit > [small refactor · elastic/elasticsearch@2cc4a28 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/2cc4a286f8ac043be4c3cf6afce2c7cc6790a4b1) then > all my data was indexed without any problems. > > To me it seems that between current master and the said commit there can be > some problem, at least that is what it looks like on my side. > > Regards, > Lukas > > On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro \< > [e.medina.m@gmail.com](mailto:e.medina.m@gmail.com)\> wrote: > > > Shay, > > > > I also have 0.16 working on my PROD env without issues, other than some GC > > delays... > > > > Thanks for the great work! > > > > On Sun, Apr 17, 2011 at 10:41 PM, ruflin [ruflin@gmail.com](mailto:ruflin@gmail.com) wrote: > > > > > Hi Shay > > > > > > I used 0.16 in the last days. I did not find any issues. 0.16 passes > > > all Elastica tests 😉 > > > > > > Thank you for all your work. I'm really looking forward to 0.16, > > > especially Tika 0.9 because I was having some troubles with the last > > > version. > > > > > > Nicolas > > > > > > On Apr 17, 8:45 pm, "K.B." [korbinian.ba...@googlemail.com](mailto:korbinian.ba...@googlemail.com) wrote: > > > > > > > Heya Shay, > > > > > > > > just wanted to say thank you for the much work you've done on that. > > > > Really appreciate it even if we had differend pov's in the past. > > > > > > > > Keep on! > > > > > > > > Best, > > > > > > > > KB > > > > > > > > On 17 Apr., 20:28, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > Heya, > > > > > > > > > 0.16 is just around the corner. Basically, master is pretty much > > > > > feature complete, so, if you have time, take it for a spin ( > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/download/master/)). > > > > > > > > > For the Java users, snapshot version have been uploaded as well. > > > > > > > > > -shay.banon ---
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [May 3, 2011, 6:42pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/15 "2011-05-03T18:42:47Z")
Today I noticed much more activity that any other days in terms of GC: [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb] [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb] [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb] [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb] [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb] [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb] [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb] Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary... On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Hi, > > I have seen some dubious messages regarding GC that ES logs lately which > seems to indicate that the API ES uses to try and log long GCs has either > changed or got broken (it uses an internal JVM API to get it). Need to look > into it. > > -shay.banon > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > Clinton, > > Thanks a lot for your interest in my GC issues... > > Out of your list, I just added: > > bootstrap: > mlockall: 1 > > to my elasticsearch.yml config file (hope the format if fine in YML), > because I already fulfilled the other hints you were suggesting: > > ulimit -l unlimited > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > Just for your evaluation, these are some samples of my GC output: > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] > [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], > leaving [255.4mb] used, max [1.1gb] > > But FYI, I've seen sometimes times in minutes... > > Regards. > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley \<[clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) > > > wrote: > > Hi Enrique > > > I also have 0.16 working on my PROD env without issues, other than > > some GC delays... > > How long are these delays? Do they cause ES to pause? > > It's quite normal to get GC messages in the logs, eg: > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], > leaving [7.1gb] used, max [14.1gb] > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving > [7.6gb] used, max [14.1gb] > > However, if any of your Java heap is sitting in swap, it can really > affect performance, and it is worth doing some work to make sure that > you never touch the swap. > > Linux can put some data into swap even when there is plenty of free > memory. > > You may already be doing this, but just in case, here is how I would set > things up: > > 1. If you have eg 8GB of memory available on your box to run ES, I would > use 50-60% of that for ES itself, and leave the rest for kernel > filesystem caches etc > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > process permission to lock memory. > > I run ES as user 'elasticsearch' and I have this line in > '/etc/security/limits.conf': > > ``` > elasticsearch - memlock unlimited > > ``` > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > the memory > > If ES is the only service you have running on the box, you may just want > to set swapoff -a > > For more, see: > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > - > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > su'ing to a different user. To enabled that, you have to uncomment this > line in /etc/pam.d/su: > > # session required pam\_limits.so > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 3, 2011, 9:12pm UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/16 "2011-05-03T21:12:17Z")
They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going. On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: Today I noticed much more activity that any other days in terms of GC: > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb] > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb] > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb] > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb] > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb] > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb] > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb] > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary... > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Hi, > > > > I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it. > > > > -shay.banon > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > > > > Clinton, > > > > > > Thanks a lot for your interest in my GC issues... > > > > > > Out of your list, I just added: > > > > > > bootstrap: > > > mlockall: 1 > > > > > > to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting: > > > > > > ulimit -l unlimited > > > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > > > > > Just for your evaluation, these are some samples of my GC output: > > > > > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb] > > > > > > But FYI, I've seen sometimes times in minutes... > > > > > > Regards. > > > > > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote: > > > > > > > Hi Enrique > > > > > > > > > I also have 0.16 working on my PROD env without issues, other than > > > > > some GC delays... > > > > > > > > How long are these delays? Do they cause ES to pause? > > > > > > > > It's quite normal to get GC messages in the logs, eg: > > > > > > > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb] > > > > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb] > > > > > > > > However, if any of your Java heap is sitting in swap, it can really > > > > affect performance, and it is worth doing some work to make sure that > > > > you never touch the swap. > > > > > > > > Linux can put some data into swap even when there is plenty of free > > > > memory. > > > > > > > > You may already be doing this, but just in case, here is how I would set > > > > things up: > > > > > > > > 1. If you have eg 8GB of memory available on your box to run ES, I would > > > > use 50-60% of that for ES itself, and leave the rest for kernel > > > > filesystem caches etc > > > > > > > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > > > > > > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > > > > process permission to lock memory. > > > > > > > > I run ES as user 'elasticsearch' and I have this line in > > > > '/etc/security/limits.conf': > > > > > > > > elasticsearch - memlock unlimited > > > > > > > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > > > > the memory > > > > > > > > If ES is the only service you have running on the box, you may just want > > > > to set swapoff -a > > > > > > > > For more, see: > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > > > > > > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > > > > su'ing to a different user. To enabled that, you have to uncomment this > > > > line in /etc/pam.d/su: > > > > > > > > # session required pam\_limits.so > > > > > > > > clint ---
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [May 4, 2011, 8:00am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/17 "2011-05-04T08:00:07Z")
Shay, Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas. What do you think could be the minimal configuration for ES to work as per my requirements? Thanks. On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > They don't take 5 minutes each, they take 5 second, the first > indicates the time it took, while the second / indicates the total time. > In any case, it seems like you might need to assign more memory for this > cluster nodes. Node stats can give you a start in trying to understand where > the memory is going. > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > Today I noticed much more activity that any other days in terms of GC: > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], > leaving [612.9mb] used, max [1gb] > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving > [842.3mb] used, max [1gb] > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], > leaving [874.7mb] used, max [1gb] > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving > [673mb] used, max [1gb] > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], > leaving [700.2mb] used, max [1gb] > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving > [718.5mb] used, max [1gb] > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], > leaving [745.1mb] used, max [1gb] > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of > 5 minutes each? It's a bit scary... > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > Hi, > > I have seen some dubious messages regarding GC that ES logs lately which > seems to indicate that the API ES uses to try and log long GCs has either > changed or got broken (it uses an internal JVM API to get it). Need to look > into it. > > -shay.banon > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > Clinton, > > Thanks a lot for your interest in my GC issues... > > Out of your list, I just added: > > bootstrap: > mlockall: 1 > > to my elasticsearch.yml config file (hope the format if fine in YML), > because I already fulfilled the other hints you were suggesting: > > ulimit -l unlimited > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > Just for your evaluation, these are some samples of my GC output: > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] > [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], > leaving [255.4mb] used, max [1.1gb] > > But FYI, I've seen sometimes times in minutes... > > Regards. > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley \<[clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) > > > wrote: > > Hi Enrique > > > I also have 0.16 working on my PROD env without issues, other than > > some GC delays... > > How long are these delays? Do they cause ES to pause? > > It's quite normal to get GC messages in the logs, eg: > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], > leaving [7.1gb] used, max [14.1gb] > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving > [7.6gb] used, max [14.1gb] > > However, if any of your Java heap is sitting in swap, it can really > affect performance, and it is worth doing some work to make sure that > you never touch the swap. > > Linux can put some data into swap even when there is plenty of free > memory. > > You may already be doing this, but just in case, here is how I would set > things up: > > 1. If you have eg 8GB of memory available on your box to run ES, I would > use 50-60% of that for ES itself, and leave the rest for kernel > filesystem caches etc > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > process permission to lock memory. > > I run ES as user 'elasticsearch' and I have this line in > '/etc/security/limits.conf': > > ``` > elasticsearch - memlock unlimited > > ``` > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > the memory > > If ES is the only service you have running on the box, you may just want > to set swapoff -a > > For more, see: > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > - > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > su'ing to a different user. To enabled that, you have to uncomment this > line in /etc/pam.d/su: > > # session required pam\_limits.so > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 4, 2011, 8:03am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/18 "2011-05-04T08:03:05Z")
If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards. -shay.banon On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: Shay, > Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas. > > What do you think could be the minimal configuration for ES to work as per my requirements? > > Thanks. > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going. > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > > > > Today I noticed much more activity that any other days in terms of GC: > > > > > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb] > > > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb] > > > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb] > > > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb] > > > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb] > > > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb] > > > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb] > > > > > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary... > > > > > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > Hi, > > > > > > > > I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it. > > > > > > > > -shay.banon > > > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > > > > > > > > Clinton, > > > > > > > > > > Thanks a lot for your interest in my GC issues... > > > > > > > > > > Out of your list, I just added: > > > > > > > > > > bootstrap: > > > > > mlockall: 1 > > > > > > > > > > to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting: > > > > > > > > > > ulimit -l unlimited > > > > > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > > > > > > > > > Just for your evaluation, these are some samples of my GC output: > > > > > > > > > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb] > > > > > > > > > > But FYI, I've seen sometimes times in minutes... > > > > > > > > > > Regards. > > > > > > > > > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote: > > > > > > > > > > > Hi Enrique > > > > > > > > > > > > > I also have 0.16 working on my PROD env without issues, other than > > > > > > > some GC delays... > > > > > > > > > > > > How long are these delays? Do they cause ES to pause? > > > > > > > > > > > > It's quite normal to get GC messages in the logs, eg: > > > > > > > > > > > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb] > > > > > > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb] > > > > > > > > > > > > However, if any of your Java heap is sitting in swap, it can really > > > > > > affect performance, and it is worth doing some work to make sure that > > > > > > you never touch the swap. > > > > > > > > > > > > Linux can put some data into swap even when there is plenty of free > > > > > > memory. > > > > > > > > > > > > You may already be doing this, but just in case, here is how I would set > > > > > > things up: > > > > > > > > > > > > 1. If you have eg 8GB of memory available on your box to run ES, I would > > > > > > use 50-60% of that for ES itself, and leave the rest for kernel > > > > > > filesystem caches etc > > > > > > > > > > > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > > > > > > > > > > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > > > > > > process permission to lock memory. > > > > > > > > > > > > I run ES as user 'elasticsearch' and I have this line in > > > > > > '/etc/security/limits.conf': > > > > > > > > > > > > elasticsearch - memlock unlimited > > > > > > > > > > > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > > > > > > the memory > > > > > > > > > > > > If ES is the only service you have running on the box, you may just want > > > > > > to set swapoff -a > > > > > > > > > > > > For more, see: > > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > > > > > > > > > > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > > > > > > su'ing to a different user. To enabled that, you have to uncomment this > > > > > > line in /etc/pam.d/su: > > > > > > > > > > > > # session required pam\_limits.so > > > > > > > > > > > > clint ---
**Author:** ![Enrique\_Medina\_Monte](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@Enrique\_Medina\_Monte](https://discuss.elastic.co/u/Enrique_Medina_Monte)\ **Post date:** [May 4, 2011, 8:09am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/19 "2011-05-04T08:09:35Z")
Then I'll go for that, but I guess I will have to re-index everything if I remove 2 shards, right? On Wed, May 4, 2011 at 10:03 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > If you have just a single node, and don't plan to expand the size of the > index, then a single shard is enough. As to replicas, if you have a single > node, then no replicas is fine (though even with 1 replica, that replica is > not allocated on the same node), and you can always increase it if you want. > A single shard will use less memory than 3 shards. > > -shay.banon > > On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: > > Shay, > > Maybe I'm overconfiguring the ES server, as I only need it to search > products by two fields. I used the defaults: 3 shards and 1 replica, but as > long as this data is not critical whatsoever (I could regenerate it in some > hours, and it doesn't stop my business), maybe I should just give it 1 shard > and no replicas. > > What do you think could be the minimal configuration for ES to work as per > my requirements? > > Thanks. > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > They don't take 5 minutes each, they take 5 second, the first > indicates the time it took, while the second / indicates the total time. > In any case, it seems like you might need to assign more memory for this > cluster nodes. Node stats can give you a start in trying to understand where > the memory is going. > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > Today I noticed much more activity that any other days in terms of GC: > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], > leaving [612.9mb] used, max [1gb] > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving > [842.3mb] used, max [1gb] > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], > leaving [874.7mb] used, max [1gb] > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving > [673mb] used, max [1gb] > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], > leaving [700.2mb] used, max [1gb] > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving > [718.5mb] used, max [1gb] > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] > [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], > leaving [745.1mb] used, max [1gb] > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of > 5 minutes each? It's a bit scary... > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > Hi, > > I have seen some dubious messages regarding GC that ES logs lately which > seems to indicate that the API ES uses to try and log long GCs has either > changed or got broken (it uses an internal JVM API to get it). Need to look > into it. > > -shay.banon > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > Clinton, > > Thanks a lot for your interest in my GC issues... > > Out of your list, I just added: > > bootstrap: > mlockall: 1 > > to my elasticsearch.yml config file (hope the format if fine in YML), > because I already fulfilled the other hints you were suggesting: > > ulimit -l unlimited > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > Just for your evaluation, these are some samples of my GC output: > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] > [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], > leaving [255.4mb] used, max [1.1gb] > > But FYI, I've seen sometimes times in minutes... > > Regards. > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley \<[clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) > > > wrote: > > Hi Enrique > > > I also have 0.16 working on my PROD env without issues, other than > > some GC delays... > > How long are these delays? Do they cause ES to pause? > > It's quite normal to get GC messages in the logs, eg: > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], > leaving [7.1gb] used, max [14.1gb] > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving > [7.6gb] used, max [14.1gb] > > However, if any of your Java heap is sitting in swap, it can really > affect performance, and it is worth doing some work to make sure that > you never touch the swap. > > Linux can put some data into swap even when there is plenty of free > memory. > > You may already be doing this, but just in case, here is how I would set > things up: > > 1. If you have eg 8GB of memory available on your box to run ES, I would > use 50-60% of that for ES itself, and leave the rest for kernel > filesystem caches etc > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > process permission to lock memory. > > I run ES as user 'elasticsearch' and I have this line in > '/etc/security/limits.conf': > > ``` > elasticsearch - memlock unlimited > > ``` > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > the memory > > If ES is the only service you have running on the box, you may just want > to set swapoff -a > > For more, see: > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > - > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > su'ing to a different user. To enabled that, you have to uncomment this > line in /etc/pam.d/su: > > # session required pam\_limits.so > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 4, 2011, 8:12am UTC](https://discuss.elastic.co/t/0-16-release-plan/4245/20 "2011-05-04T08:12:04Z")
Yes, you will need to. But, I would also check what takes that memory. Node stats can give some info. Another place that takes memory is the internal data structures in Lucene itself to support searching, but those are not exposed as stats. On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote: > Then I'll go for that, but I guess I will have to re-index everything if I remove 2 shards, right? > > On Wed, May 4, 2011 at 10:03 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards. > > > > -shay.banon > > On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote: > > > > > Shay, > > > > > > Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas. > > > > > > What do you think could be the minimal configuration for ES to work as per my requirements? > > > > > > Thanks. > > > > > > On Tue, May 3, 2011 at 11:12 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going. > > > > On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote: > > > > > > > > > Today I noticed much more activity that any other days in terms of GC: > > > > > > > > > > [2011-05-02 12:20:17,327][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb] > > > > > [2011-05-02 12:21:02,395][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb] > > > > > [2011-05-02 12:21:16,026][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb] > > > > > [2011-05-02 12:22:12,922][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb] > > > > > [2011-05-02 12:22:19,094][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb] > > > > > [2011-05-02 12:22:26,582][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb] > > > > > [2011-05-02 12:22:35,417][INFO][monitor.jvm] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb] > > > > > > > > > > Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary... > > > > > > > > > > On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it. > > > > > > > > > > > > -shay.banon > > > > > > On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote: > > > > > > > > > > > > > Clinton, > > > > > > > > > > > > > > Thanks a lot for your interest in my GC issues... > > > > > > > > > > > > > > Out of your list, I just added: > > > > > > > > > > > > > > bootstrap: > > > > > > > mlockall: 1 > > > > > > > > > > > > > > to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting: > > > > > > > > > > > > > > ulimit -l unlimited > > > > > > > /var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g & > > > > > > > > > > > > > > Just for your evaluation, these are some samples of my GC output: > > > > > > > > > > > > > > [2011-04-18 10:43:12,001][INFO][monitor.jvm] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb] > > > > > > > > > > > > > > But FYI, I've seen sometimes times in minutes... > > > > > > > > > > > > > > Regards. > > > > > > > > > > > > > > On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote: > > > > > > > > > > > > > > > Hi Enrique > > > > > > > > > > > > > > > > > I also have 0.16 working on my PROD env without issues, other than > > > > > > > > > some GC delays... > > > > > > > > > > > > > > > > How long are these delays? Do they cause ES to pause? > > > > > > > > > > > > > > > > It's quite normal to get GC messages in the logs, eg: > > > > > > > > > > > > > > > > [gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb] > > > > > > > > [gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb] > > > > > > > > > > > > > > > > However, if any of your Java heap is sitting in swap, it can really > > > > > > > > affect performance, and it is worth doing some work to make sure that > > > > > > > > you never touch the swap. > > > > > > > > > > > > > > > > Linux can put some data into swap even when there is plenty of free > > > > > > > > memory. > > > > > > > > > > > > > > > > You may already be doing this, but just in case, here is how I would set > > > > > > > > things up: > > > > > > > > > > > > > > > > 1. If you have eg 8GB of memory available on your box to run ES, I would > > > > > > > > use 50-60% of that for ES itself, and leave the rest for kernel > > > > > > > > filesystem caches etc > > > > > > > > > > > > > > > > 2. Set your ES\_MIN\_MEM and ES\_MAX\_MEM to the same value (eg 4GB) > > > > > > > > > > > > > > > > 3. Set "ulimit -l unlimited" before starting ES - this gives the ES > > > > > > > > process permission to lock memory. > > > > > > > > > > > > > > > > I run ES as user 'elasticsearch' and I have this line in > > > > > > > > '/etc/security/limits.conf': > > > > > > > > > > > > > > > > elasticsearch - memlock unlimited > > > > > > > > > > > > > > > > 1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock > > > > > > > > the memory > > > > > > > > > > > > > > > > If ES is the only service you have running on the box, you may just want > > > > > > > > to set swapoff -a > > > > > > > > > > > > > > > > For more, see: > > > > > > > > > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html) > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2010/07/01/setting-up-elasticsearch.html) > > > > > > > > - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/tutorials/2011/02/22/running-elasticsearch-as-a-non-root-user.html) > > > > > > > > > > > > > > > > Note: Ubuntu doesn't read /etc/security/limits.conf by default, when > > > > > > > > su'ing to a different user. To enabled that, you have to uncomment this > > > > > > > > line in /etc/pam.d/su: > > > > > > > > > > > > > > > > # session required pam\_limits.so > > > > > > > > > > > > > > > > clint [Next page](https://discuss.elastic.co/t/0-16-release-plan/4245.md?page=2)