0.16 Release Plan

Elastic Stack Elasticsearch
21

Thanks Shay.

On Wed, May 4, 2011 at 10:12 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Yes, you will need to. But, I would also check what takes that memory. Node
stats can give some info. Another place that takes memory is the internal
data structures in Lucene itself to support searching, but those are not
exposed as stats.

On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote:

Then I'll go for that, but I guess I will have to re-index everything if I
remove 2 shards, right?

On Wed, May 4, 2011 at 10:03 AM, Shay Banon shay.banon@elasticsearch.comwrote:

If you have just a single node, and don't plan to expand the size of the
index, then a single shard is enough. As to replicas, if you have a single
node, then no replicas is fine (though even with 1 replica, that replica is
not allocated on the same node), and you can always increase it if you want.
A single shard will use less memory than 3 shards.

-shay.banon

On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search
products by two fields. I used the defaults: 3 shards and 1 replica, but as
long as this data is not critical whatsoever (I could regenerate it in some
hours, and it doesn't stop my business), maybe I should just give it 1 shard
and no replicas.

What do you think could be the minimal configuration for ES to work as per
my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.comwrote:

They don't take 5 minutes each, they take 5 second, the first
indicates the time it took, while the second / indicates the total time.
In any case, it seems like you might need to assign more memory for this
cluster nodes. Node stats can give you a start in trying to understand where
the memory is going.

On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb],
leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving
[842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb],
leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving
[673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb],
leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving
[718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb],
leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of
5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which
seems to indicate that the API ES uses to try and log long GCs has either
changed or got broken (it uses an internal JVM API to get it). Need to look
into it.

-shay.banon

On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML),
because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake]
[gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb],
leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley <clinton@iannounce.co.uk

wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb],
leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving
[7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

  elasticsearch     -    memlock    unlimited
  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

22

These are my node stats:

Where should I focus to analyze GCs?

On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro <
e.medina.m@gmail.com> wrote:

Thanks Shay.

On Wed, May 4, 2011 at 10:12 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Yes, you will need to. But, I would also check what takes that memory.
Node stats can give some info. Another place that takes memory is the
internal data structures in Lucene itself to support searching, but those
are not exposed as stats.

On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote:

Then I'll go for that, but I guess I will have to re-index everything if I
remove 2 shards, right?

On Wed, May 4, 2011 at 10:03 AM, Shay Banon <shay.banon@elasticsearch.com

wrote:

If you have just a single node, and don't plan to expand the size of the
index, then a single shard is enough. As to replicas, if you have a single
node, then no replicas is fine (though even with 1 replica, that replica is
not allocated on the same node), and you can always increase it if you want.
A single shard will use less memory than 3 shards.

-shay.banon

On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search
products by two fields. I used the defaults: 3 shards and 1 replica, but as
long as this data is not critical whatsoever (I could regenerate it in some
hours, and it doesn't stop my business), maybe I should just give it 1 shard
and no replicas.

What do you think could be the minimal configuration for ES to work as per
my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

They don't take 5 minutes each, they take 5 second, the first
indicates the time it took, while the second / indicates the total time.
In any case, it seems like you might need to assign more memory for this
cluster nodes. Node stats can give you a start in trying to understand where
the memory is going.

On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb],
leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving
[842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb],
leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving
[673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb],
leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving
[718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb],
leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of
5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately
which seems to indicate that the API ES uses to try and log long GCs has
either changed or got broken (it uses an internal JVM API to get it). Need
to look into it.

-shay.banon

On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML),
because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake]
[gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb],
leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley <
clinton@iannounce.co.uk> wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb],
leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving
[7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

  elasticsearch     -    memlock    unlimited
  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

23

Its less a question of the GC, it more to see what takes memory, and if you might need to increase the memory simply because of the way you use the system. It seems like hte field cache is ~300mb, which is substantial when allocating 1gb to the ES process.
On Wednesday, May 4, 2011 at 12:15 PM, Enrique Medina Montenegro wrote:

These are my node stats:

gist:954972 · GitHub

Where should I focus to analyze GCs?

On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro e.medina.m@gmail.com wrote:

Thanks Shay.

On Wed, May 4, 2011 at 10:12 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Yes, you will need to. But, I would also check what takes that memory. Node stats can give some info. Another place that takes memory is the internal data structures in Lucene itself to support searching, but those are not exposed as stats.
On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote:

Then I'll go for that, but I guess I will have to re-index everything if I remove 2 shards, right?

On Wed, May 4, 2011 at 10:03 AM, Shay Banon shay.banon@elasticsearch.com wrote:

If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards.

-shay.banon
On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas.

What do you think could be the minimal configuration for ES to work as per my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.com wrote:

They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going.
On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it.

-shay.banon
On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

elasticsearch - memlock unlimited

  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

24

Can I set the max_size for field cache? Where exactly?

On Wed, May 4, 2011 at 6:38 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Its less a question of the GC, it more to see what takes memory, and if
you might need to increase the memory simply because of the way you use the
system. It seems like hte field cache is ~300mb, which is substantial when
allocating 1gb to the ES process.

On Wednesday, May 4, 2011 at 12:15 PM, Enrique Medina Montenegro wrote:

These are my node stats:

gist:954972 · GitHub

Where should I focus to analyze GCs?

On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro <
e.medina.m@gmail.com> wrote:

Thanks Shay.

On Wed, May 4, 2011 at 10:12 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Yes, you will need to. But, I would also check what takes that memory. Node
stats can give some info. Another place that takes memory is the internal
data structures in Lucene itself to support searching, but those are not
exposed as stats.

On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote:

Then I'll go for that, but I guess I will have to re-index everything if I
remove 2 shards, right?

On Wed, May 4, 2011 at 10:03 AM, Shay Banon shay.banon@elasticsearch.comwrote:

If you have just a single node, and don't plan to expand the size of the
index, then a single shard is enough. As to replicas, if you have a single
node, then no replicas is fine (though even with 1 replica, that replica is
not allocated on the same node), and you can always increase it if you want.
A single shard will use less memory than 3 shards.

-shay.banon

On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search
products by two fields. I used the defaults: 3 shards and 1 replica, but as
long as this data is not critical whatsoever (I could regenerate it in some
hours, and it doesn't stop my business), maybe I should just give it 1 shard
and no replicas.

What do you think could be the minimal configuration for ES to work as per
my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.comwrote:

They don't take 5 minutes each, they take 5 second, the first
indicates the time it took, while the second / indicates the total time.
In any case, it seems like you might need to assign more memory for this
cluster nodes. Node stats can give you a start in trying to understand where
the memory is going.

On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb],
leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving
[842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb],
leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving
[673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb],
leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving
[718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind]
[gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb],
leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of
5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which
seems to indicate that the API ES uses to try and log long GCs has either
changed or got broken (it uses an internal JVM API to get it). Need to look
into it.

-shay.banon

On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML),
because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake]
[gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb],
leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley <clinton@iannounce.co.uk

wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb],
leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving
[7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

  elasticsearch     -    memlock    unlimited
  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint

25

Its not recommended to set it, since the field cache is a heavy one, that can potentially take time to recreate. So, if you are going to use it on a specific field, you want to keep it loaded and not unload it. Field cache is used when sorting on a field or faceting on it.
On Thursday, May 5, 2011 at 1:32 PM, Enrique Medina Montenegro wrote:
Can I set the max_size for field cache? Where exactly?

On Wed, May 4, 2011 at 6:38 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Its less a question of the GC, it more to see what takes memory, and if you might need to increase the memory simply because of the way you use the system. It seems like hte field cache is ~300mb, which is substantial when allocating 1gb to the ES process.
On Wednesday, May 4, 2011 at 12:15 PM, Enrique Medina Montenegro wrote:

These are my node stats:

gist:954972 · GitHub

Where should I focus to analyze GCs?

On Wed, May 4, 2011 at 10:30 AM, Enrique Medina Montenegro e.medina.m@gmail.com wrote:

Thanks Shay.

On Wed, May 4, 2011 at 10:12 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Yes, you will need to. But, I would also check what takes that memory. Node stats can give some info. Another place that takes memory is the internal data structures in Lucene itself to support searching, but those are not exposed as stats.
On Wednesday, May 4, 2011 at 11:09 AM, Enrique Medina Montenegro wrote:

Then I'll go for that, but I guess I will have to re-index everything if I remove 2 shards, right?

On Wed, May 4, 2011 at 10:03 AM, Shay Banon shay.banon@elasticsearch.com wrote:

If you have just a single node, and don't plan to expand the size of the index, then a single shard is enough. As to replicas, if you have a single node, then no replicas is fine (though even with 1 replica, that replica is not allocated on the same node), and you can always increase it if you want. A single shard will use less memory than 3 shards.

-shay.banon
On Wednesday, May 4, 2011 at 11:00 AM, Enrique Medina Montenegro wrote:

Shay,

Maybe I'm overconfiguring the ES server, as I only need it to search products by two fields. I used the defaults: 3 shards and 1 replica, but as long as this data is not critical whatsoever (I could regenerate it in some hours, and it doesn't stop my business), maybe I should just give it 1 shard and no replicas.

What do you think could be the minimal configuration for ES to work as per my requirements?

Thanks.

On Tue, May 3, 2011 at 11:12 PM, Shay Banon shay.banon@elasticsearch.com wrote:

They don't take 5 minutes each, they take 5 second, the first indicates the time it took, while the second / indicates the total time. In any case, it seems like you might need to assign more memory for this cluster nodes. Node stats can give you a start in trying to understand where the memory is going.
On Tuesday, May 3, 2011 at 9:42 PM, Enrique Medina Montenegro wrote:

Today I noticed much more activity that any other days in terms of GC:

[2011-05-02 12:20:17,327][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][347] took [7.9s]/[3.7m], reclaimed [17.7mb], leaving [612.9mb] used, max [1gb]
[2011-05-02 12:21:02,395][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][354] took [5s]/[4.2m], reclaimed [8.4mb], leaving [842.3mb] used, max [1gb]
[2011-05-02 12:21:16,026][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][356] took [5.7s]/[4.4m], reclaimed [14.9mb], leaving [874.7mb] used, max [1gb]
[2011-05-02 12:22:12,922][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][365] took [5s]/[5m], reclaimed [20.1mb], leaving [673mb] used, max [1gb]
[2011-05-02 12:22:19,094][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][366] took [5.4s]/[5.1m], reclaimed [18.1mb], leaving [700.2mb] used, max [1gb]
[2011-05-02 12:22:26,582][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][367] took [5s]/[5.2m], reclaimed [19mb], leaving [718.5mb] used, max [1gb]
[2011-05-02 12:22:35,417][INFO ][monitor.jvm ] [Uni-Mind] [gc][ConcurrentMarkSweep][368] took [5.1s]/[5.3m], reclaimed [14.6mb], leaving [745.1mb] used, max [1gb]

Does it make sense to perform 7 consecutive GCs? And lasting an avergae of 5 minutes each? It's a bit scary...

On Tue, Apr 19, 2011 at 11:04 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

I have seen some dubious messages regarding GC that ES logs lately which seems to indicate that the API ES uses to try and log long GCs has either changed or got broken (it uses an internal JVM API to get it). Need to look into it.

-shay.banon
On Monday, April 18, 2011 at 1:00 PM, Enrique Medina Montenegro wrote:

Clinton,

Thanks a lot for your interest in my GC issues...

Out of your list, I just added:

bootstrap:
mlockall: 1

to my elasticsearch.yml config file (hope the format if fine in YML), because I already fulfilled the other hints you were suggesting:

ulimit -l unlimited
/var/elasticsearch-0.16.0-SNAPSHOT/bin/elasticsearch -f -Xmx1g -Xms1g &

Just for your evaluation, these are some samples of my GC output:

[2011-04-18 10:43:12,001][INFO ][monitor.jvm ] [Earthquake] [gc][ConcurrentMarkSweep][2] took [27.5s]/[27.7s], reclaimed [532.8mb], leaving [255.4mb] used, max [1.1gb]

But FYI, I've seen sometimes times in minutes...

Regards.

On Mon, Apr 18, 2011 at 11:28 AM, Clinton Gormley clinton@iannounce.co.uk wrote:

Hi Enrique

I also have 0.16 working on my PROD env without issues, other than
some GC delays...

How long are these delays? Do they cause ES to pause?

It's quite normal to get GC messages in the logs, eg:

[gc][ConcurrentMarkSweep][23] took [12.2s]/[2.9s], reclaimed [3.2gb], leaving [7.1gb] used, max [14.1gb]
[gc][ConcurrentMarkSweep][24] took [10s]/[3s], reclaimed [2.7gb], leaving [7.6gb] used, max [14.1gb]

However, if any of your Java heap is sitting in swap, it can really
affect performance, and it is worth doing some work to make sure that
you never touch the swap.

Linux can put some data into swap even when there is plenty of free
memory.

You may already be doing this, but just in case, here is how I would set
things up:

  1. If you have eg 8GB of memory available on your box to run ES, I would
    use 50-60% of that for ES itself, and leave the rest for kernel
    filesystem caches etc

  2. Set your ES_MIN_MEM and ES_MAX_MEM to the same value (eg 4GB)

  3. Set "ulimit -l unlimited" before starting ES - this gives the ES
    process permission to lock memory.

I run ES as user 'elasticsearch' and I have this line in
'/etc/security/limits.conf':

elasticsearch - memlock unlimited

  1. Add "boostrap.mlockall: 1" to your config file, to tell ES to lock
    the memory

If ES is the only service you have running on the box, you may just want
to set swapoff -a

For more, see:

Note: Ubuntu doesn't read /etc/security/limits.conf by default, when
su'ing to a different user. To enabled that, you have to uncomment this
line in /etc/pam.d/su:

session required pam_limits.so

clint