0.16 Release Plan

Elastic Stack Elasticsearch
1

Heya,

0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin (http://www.elasticsearch.org/download/master/).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

2

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin (Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

3

Hi Shay

I used 0.16 in the last days. I did not find any issues. 0.16 passes
all Elastica tests :wink:

Thank you for all your work. I'm really looking forward to 0.16,
especially Tika 0.9 because I was having some troubles with the last
version.

Nicolas

On Apr 17, 8:45 pm, "K.B." korbinian.ba...@googlemail.com wrote:

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin (Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

4

Shay,

I also have 0.16 working on my PROD env without issues, other than some GC
delays...

Thanks for the great work!

On Sun, Apr 17, 2011 at 10:41 PM, ruflin ruflin@gmail.com wrote:

Hi Shay

I used 0.16 in the last days. I did not find any issues. 0.16 passes
all Elastica tests :wink:

Thank you for all your work. I'm really looking forward to 0.16,
especially Tika 0.9 because I was having some troubles with the last
version.

Nicolas

On Apr 17, 8:45 pm, "K.B." korbinian.ba...@googlemail.com wrote:

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much
feature complete, so, if you have time, take it for a spin (
Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

5

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

    I run ES as user 'elasticsearch' and I have this line in
    '/etc/security/limits.conf':

    elasticsearch     -    memlock    unlimited

  4. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

# session    required   pam_limits.so

clint

6

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML),
because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake]
[gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb],
leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley
clinton@iannounce.co.ukwrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb],
leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving
[7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

  elasticsearch     -    memlock    unlimited
  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

7

Hi,

I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it.

-shay.banon
On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

elasticsearch - memlock unlimited

  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

8

Hi,

Hate to tell this but I encountered some issues. I upgraded to the recent
master and found out that I got hanging thread during indexing (I do not
have a rigorous test for this, I just run indexing of data manually and 3x
in row I got some hanging thread, this can be coincidence but if it is then
I would say quite rare). On the other hand when I downgraded to commit

then
all my data was indexed without any problems.

To me it seems that between current master and the said commit there can be
some problem, at least that is what it looks like on my side.

Regards,
Lukas

On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro <
e.medina.m@gmail.com> wrote:

Shay,

I also have 0.16 working on my PROD env without issues, other than some GC
delays...

Thanks for the great work!

On Sun, Apr 17, 2011 at 10:41 PM, ruflin ruflin@gmail.com wrote:

Hi Shay

I used 0.16 in the last days. I did not find any issues. 0.16 passes
all Elastica tests :wink:

Thank you for all your work. I'm really looking forward to 0.16,
especially Tika 0.9 because I was having some troubles with the last
version.

Nicolas

On Apr 17, 8:45 pm, "K.B." korbinian.ba...@googlemail.com wrote:

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much
feature complete, so, if you have time, take it for a spin (
Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

9

Recreation?
On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote:

Hi,

Hate to tell this but I encountered some issues. I upgraded to the recent master and found out that I got hanging thread during indexing (I do not have a rigorous test for this, I just run indexing of data manually and 3x in row I got some hanging thread, this can be coincidence but if it is then I would say quite rare). On the other hand when I downgraded to commit small refactor · elastic/elasticsearch@2cc4a28 · GitHub then all my data was indexed without any problems.

To me it seems that between current master and the said commit there can be some problem, at least that is what it looks like on my side.

Regards,
Lukas

On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro e.medina.m@gmail.com wrote:

Shay,

I also have 0.16 working on my PROD env without issues, other than some GC delays...

Thanks for the great work!

On Sun, Apr 17, 2011 at 10:41 PM, ruflin ruflin@gmail.com wrote:

Hi Shay

I used 0.16 in the last days. I did not find any issues. 0.16 passes
all Elastica tests :wink:

Thank you for all your work. I'm really looking forward to 0.16,
especially Tika 0.9 because I was having some troubles with the last
version.

Nicolas

On Apr 17, 8:45 pm, "K.B." korbinian.ba...@googlemail.com wrote:

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin (Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

10

Yea I know.... probably not easy for me now... at least I will try to
identify the commit that "breaks" it for me.

On Tue, Apr 19, 2011 at 11:15 AM, Shay Banon
shay.banon@elasticsearch.comwrote:

Recreation?

On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote:

Hi,

Hate to tell this but I encountered some issues. I upgraded to the recent
master and found out that I got hanging thread during indexing (I do not
have a rigorous test for this, I just run indexing of data manually and 3x
in row I got some hanging thread, this can be coincidence but if it is then
I would say quite rare). On the other hand when I downgraded to commit
small refactor · elastic/elasticsearch@2cc4a28 · GitHub then
all my data was indexed without any problems.

To me it seems that between current master and the said commit there can be
some problem, at least that is what it looks like on my side.

Regards,
Lukas

On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro <
e.medina.m@gmail.com> wrote:

Shay,

I also have 0.16 working on my PROD env without issues, other than some GC
delays...

Thanks for the great work!

On Sun, Apr 17, 2011 at 10:41 PM, ruflin ruflin@gmail.com wrote:

Hi Shay

I used 0.16 in the last days. I did not find any issues. 0.16 passes
all Elastica tests :wink:

Thank you for all your work. I'm really looking forward to 0.16,
especially Tika 0.9 because I was having some troubles with the last
version.

Nicolas

On Apr 17, 8:45 pm, "K.B." korbinian.ba...@googlemail.com wrote:

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much
feature complete, so, if you have time, take it for a spin (
Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

11 
Any progress on the release? 


I am about to set up my production environment now and I was hoping
to jump on the final 0.16 version. But if there are still problems I
can wait and use the 0.15.2 version. Is there any substantial
differences in performance or memory usage between 0.15.2 and 0.16
that may be of interest or is it just functional upgrades (which I
do not need right now)? 




Main question, jump on 0.16 master and build, or stick with 0.15.2?




/Kristian




Lukáš Vlček skrev 2011-04-19 11:19:
<blockquote cite="mid:BANLkTikBtA4tu7XfE7NSy+vi78n1nVtFAw@mail.gmail.com" type="cite">Yea I know.... probably not easy for me now... at
  least I will try to identify the commit that "breaks" it for me.

On Tue, Apr 19, 2011 at 11:15 AM, Shay
Banon <shay.banon@elasticsearch.com>
wrote:

Recreation?

On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote:

Hi,

Hate to tell this but I encountered some
issues. I upgraded to the recent master and
found out that I got hanging thread during
indexing (I do not have a rigorous test for
this, I just run indexing of data manually
and 3x in row I got some hanging thread,
this can be coincidence but if it is then I
would say quite rare). On the other hand
when I downgraded to commit https://github.com/elasticsearch/elasticsearch/commit/2cc4a286f8ac043be4c3cf6afce2c7cc6790a4b1 then
all my data was indexed without any
problems.

To me it seems that between current
master and the said commit there can be some
problem, at least that is what it looks like
on my side.

Regards,Lukas

                          On Mon, Apr 18, 2011 at 11:09 AM, Enrique
                          Medina Montenegro &lt;<a moz-do-not-send="true" href="mailto:e.medina.m@gmail.com" target="_blank">e.medina.m@gmail.com</a>&gt;
                          wrote:
Shay,

I also have 0.16 working on my
PROD env without issues, other than
some GC delays...

                                Thanks for the great work!
                                




                                      On Sun, Apr 17, 2011 at 10:41
                                      PM, ruflin &lt;<a moz-do-not-send="true" href="mailto:ruflin@gmail.com" target="_blank">ruflin@gmail.com</a>&gt;
                                      wrote:
Hi Shay 
                                          I used 0.16 in the last
                                          days. I did not find any
                                          issues. 0.16 passes


                                          all Elastica tests ;-)




                                          Thank you for all your
                                          work. I'm really looking
                                          forward to 0.16,


                                          especially Tika 0.9
                                          because I was having some
                                          troubles with the last


                                          version.




                                          Nicolas




                                              On Apr 17, 8:45Â pm,
                                              "K.B." &lt;<a moz-do-not-send="true" href="mailto:korbinian.ba...@googlemail.com" target="_blank">korbinian.ba...@googlemail.com</a>&gt;
                                              wrote:


                                              &gt; Heya Shay,


                                              &gt;


                                              &gt; just wanted to
                                              say thank you for the
                                              much work you've done
                                              on that.


                                              &gt; Really appreciate
                                              it even if we had
                                              differend pov's in the
                                              past.


                                              &gt;


                                              &gt; Keep on!


                                              &gt;


                                              &gt; Best,


                                              &gt;


                                              &gt; KB


                                              &gt;


                                              &gt; On 17 Apr.,
                                              20:28, Shay Banon &lt;<a moz-do-not-send="true" href="mailto:shay.ba...@elasticsearch.com" target="_blank">shay.ba...@elasticsearch.com</a>&gt;
                                              wrote:


                                              &gt;


                                              &gt;


                                              &gt;


                                              &gt; &gt; Heya,


                                              &gt;


                                              &gt; &gt; 0.16 is just
                                              around the corner.
                                              Basically, master is
                                              pretty much feature
                                              complete, so, if you
                                              have time, take it for
                                              a spin (<a moz-do-not-send="true" href="http://www.elasticsearch.org/download/master/" target="_blank">http://www.elasticsearch.org/download/master/</a>).


                                              &gt;


                                              &gt; &gt; For the Java
                                              users, snapshot
                                              version have been
                                              uploaded as well.


                                              &gt;


                                              &gt; &gt; -shay.banon

-- 
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post: kristian.jorg@devo.se
Webb: http://www.devo.se
12

Current plan is to release 0.16 either today or tomorrow (still checking if a couple of small features will make it in, and some small bug fixes open).

There are a lot of improvements in 0.16, but, don't push it to prod before you tested it with your app.

-shay.banon
On Thursday, April 21, 2011 at 1:34 PM, Kristian Jörg wrote:

Any progress on the release?
I am about to set up my production environment now and I was hoping to jump on the final 0.16 version. But if there are still problems I can wait and use the 0.15.2 version. Is there any substantial differences in performance or memory usage between 0.15.2 and 0.16 that may be of interest or is it just functional upgrades (which I do not need right now)?

Main question, jump on 0.16 master and build, or stick with 0.15.2?

/Kristian

Lukáš Vlček skrev 2011-04-19 11:19:

Yea I know.... probably not easy for me now... at least I will try to identify the commit that "breaks" it for me.

On Tue, Apr 19, 2011 at 11:15 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Recreation?
On Tuesday, April 19, 2011 at 12:11 PM, Lukáš Vlček wrote:

Hi,

Hate to tell this but I encountered some issues. I upgraded to the recent master and found out that I got hanging thread during indexing (I do not have a rigorous test for this, I just run indexing of data manually and 3x in row I got some hanging thread, this can be coincidence but if it is then I would say quite rare). On the other hand when I downgraded to commit small refactor · elastic/elasticsearch@2cc4a28 · GitHub then all my data was indexed without any problems.

To me it seems that between current master and the said commit there can be some problem, at least that is what it looks like on my side.

Regards,
Lukas

On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro e.medina.m@gmail.com wrote:

Shay,

I also have 0.16 working on my PROD env without issues, other than some GC delays...

Thanks for the great work!

On Sun, Apr 17, 2011 at 10:41 PM, ruflin ruflin@gmail.com wrote:

Hi Shay

I used 0.16 in the last days. I did not find any issues. 0.16 passes
all Elastica tests :wink:

Thank you for all your work. I'm really looking forward to 0.16,
especially Tika 0.9 because I was having some troubles with the last
version.

Nicolas

On Apr 17, 8:45 pm, "K.B." korbinian.ba...@googlemail.com wrote:

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much feature complete, so, if you have time, take it for a spin (Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

-- Med vänlig hälsning Kristian Jörg Devo IT AB Tel: 054 - 22 14 58, 0709 - 15 83 42 E-post: kristian.jorg@devo.se Webb: http://www.devo.se

13

Shay Banon skrev 2011-04-21 12:40:

Current plan is to release 0.16 either today or tomorrow (still
checking if a couple of small features will make it in, and some small
bug fixes open).

Thanks for the update.
Yes, I will start prod on 0.15.2 and test 0.16 befor upgrading, I came
to that conclusion myself after a cup of coffee. :slight_smile:

Regards
Kristian

14

Hi,

just follow up on my "hanging thread" issue, the problem was on my side. I
did not update all the elaticsearch files and sources correctly on the
server. It runs fine for me now.

Thanks,
Lukas

On Tue, Apr 19, 2011 at 11:11 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

Hate to tell this but I encountered some issues. I upgraded to the recent
master and found out that I got hanging thread during indexing (I do not
have a rigorous test for this, I just run indexing of data manually and 3x
in row I got some hanging thread, this can be coincidence but if it is then
I would say quite rare). On the other hand when I downgraded to commit
small refactor · elastic/elasticsearch@2cc4a28 · GitHub then
all my data was indexed without any problems.

To me it seems that between current master and the said commit there can be
some problem, at least that is what it looks like on my side.

Regards,
Lukas

On Mon, Apr 18, 2011 at 11:09 AM, Enrique Medina Montenegro <
e.medina.m@gmail.com> wrote:

Shay,

I also have 0.16 working on my PROD env without issues, other than some GC
delays...

Thanks for the great work!

On Sun, Apr 17, 2011 at 10:41 PM, ruflin ruflin@gmail.com wrote:

Hi Shay

I used 0.16 in the last days. I did not find any issues. 0.16 passes
all Elastica tests :wink:

Thank you for all your work. I'm really looking forward to 0.16,
especially Tika 0.9 because I was having some troubles with the last
version.

Nicolas

On Apr 17, 8:45 pm, "K.B." korbinian.ba...@googlemail.com wrote:

Heya Shay,

just wanted to say thank you for the much work you've done on that.
Really appreciate it even if we had differend pov's in the past.

Keep on!

Best,

KB

On 17 Apr., 20:28, Shay Banon shay.ba...@elasticsearch.com wrote:

Heya,

0.16 is just around the corner. Basically, master is pretty much
feature complete, so, if you have time, take it for a spin (
Elasticsearch Platform — Find real-time answers at scale | Elastic).

For the Java users, snapshot version have been uploaded as well.

-shay.banon

15

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb],
leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving
[842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb],
leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving
[673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb],
leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving
[718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb],
leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5
minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon
shay.banon@elasticsearch.comwrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which
seems to indicate that the API ES uses to try and log long GCs has either
changed or got broken (it uses an internal JVM API to get it). Need to look
into it.

-shay.banon

On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML),
because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake]
[gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb],
leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley <clinton@iannounce.co.uk

wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb],
leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving
[7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

  elasticsearch     -    memlock    unlimited
  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

16

They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going.
On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:
Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it.

-shay.banon
On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

elasticsearch - memlock unlimited

  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

17

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search
products by two fields. I used the defaults: 3 shards and 1 replica, but as
long as this data is not critical whatsoever (I could regenerate it in some
hours, and it doesn't stop my business), maybe I should just give it 1 shard
and no replicas.

What do you think could be the minimal configuration for ES to work as per
my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.comwrote:

They don't take 5 minutes each, they take 5 second, the first
indicates the time it took, while the second / indicates the total time.
In any case, it seems like you might need to assign more memory for this
cluster nodes. Node stats can give you a start in trying to understand where
the memory is going.

On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb],
leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving
[842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb],
leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving
[673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb],
leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving
[718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb],
leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of
5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which
seems to indicate that the API ES uses to try and log long GCs has either
changed or got broken (it uses an internal JVM API to get it). Need to look
into it.

-shay.banon

On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML),
because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake]
[gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb],
leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley <clinton@iannounce.co.uk

wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb],
leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving
[7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

  elasticsearch     -    memlock    unlimited
  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

18

If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards.

-shay.banon
On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:
Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas.

What do you think could be the minimal configuration for ES to work as per my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.com wrote:

They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going.
On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it.

-shay.banon
On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

elasticsearch - memlock unlimited

  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

19

Then I'll go for that, but I guess I will have to re-index everything if I
remove 2 shards, right?

On Wed, May 4, 2011 at 10:03 AM, Shay Banon shay.banon@elasticsearch.comwrote:

If you have just a single node, and don't plan to expand the size of the
index, then a single shard is enough. As to replicas, if you have a single
node, then no replicas is fine (though even with 1 replica, that replica is
not allocated on the same node), and you can always increase it if you want.
A single shard will use less memory than 3 shards.

-shay.banon

On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search
products by two fields. I used the defaults: 3 shards and 1 replica, but as
long as this data is not critical whatsoever (I could regenerate it in some
hours, and it doesn't stop my business), maybe I should just give it 1 shard
and no replicas.

What do you think could be the minimal configuration for ES to work as per
my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.comwrote:

They don't take 5 minutes each, they take 5 second, the first
indicates the time it took, while the second / indicates the total time.
In any case, it seems like you might need to assign more memory for this
cluster nodes. Node stats can give you a start in trying to understand where
the memory is going.

On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb],
leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving
[842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb],
leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving
[673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb],
leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving
[718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb],
leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of
5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which
seems to indicate that the API ES uses to try and log long GCs has either
changed or got broken (it uses an internal JVM API to get it). Need to look
into it.

-shay.banon

On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML),
because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake]
[gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb],
leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley <clinton@iannounce.co.uk

wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb],
leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving
[7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

  elasticsearch     -    memlock    unlimited
  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

20

Yes, you will need to. But, I would also check what takes that memory. Node stats can give some info. Another place that takes memory is the internal data structures in Lucene itself to support searching, but those are not exposed as stats.
On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote:

Then I'll go for that, but I guess I will have to re-index everything if I remove 2 shards, right?

On Wed, May 4, 2011 at 10:03 AM, Shay Banon shay.banon@elasticsearch.com wrote:

If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards.

-shay.banon
On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas.

What do you think could be the minimal configuration for ES to work as per my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.com wrote:

They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going.
On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it.

-shay.banon
On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

elasticsearch - memlock unlimited

  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint