ElasticSearch 0.90.10 : GC does not free memory

Elastic Stack Elasticsearch
1

Hi everybody,

I have a problem with the latest version of ElasticSearch (0.90.10).
It seems that whatever I do, after a little while (some minutes) the
garbage collector does not free any memory.

Here are some of the logs:
[2014-01-24 09:52:05,809][WARN ][monitor.jvm ] [Walter White]
[gc][old][806][1] duration [7.2s], collections [1]/[7.4s], total
[7.2s]/[7.2s], memory [9.7gb]->[8.5gb]/[10gb], all_pools {[young]
[440mb]->[0b]/[0b]}{[survivor] [64mb]->[0b]/[0b]}{[old]
[9.3gb]->[8.5gb]/[10gb]}
[2014-01-24 09:52:52,274][WARN ][monitor.jvm ] [Walter White]
[gc][old][848][2] duration [4.5s], collections [1]/[5.4s], total
[4.5s]/[11.7s], memory [9.4gb]->[9.3gb]/[10gb], all_pools {[young]
[428mb]->[0b]/[0b]}{[survivor] [64mb]->[0b]/[0b]}{[old]
[8.9gb]->[9.3gb]/[10gb]}
[2014-01-24 09:53:08,454][WARN ][monitor.jvm ] [Walter White]
[gc][old][861][3] duration [4s], collections [1]/[4.1s], total
[4s]/[15.7s], memory [9.6gb]->[9.6gb]/[10gb], all_pools {[young]
[280mb]->[0b]/[0b]}{[survivor] [0b]->[0b]/[0b]}{[old]
[9.3gb]->[9.6gb]/[10gb]}
[2014-01-24 09:53:12,753][WARN ][monitor.jvm ] [Walter White]
[gc][old][862][4] duration [3.9s], collections [1]/[4.2s], total
[3.9s]/[19.6s], memory [9.6gb]->[9.6gb]/[10gb], all_pools {[young]
[0b]->[0b]/[0b]}{[survivor] [0b]->[0b]/[0b]}{[old] [9.6gb]->[9.6gb]/[10gb]}

We clearly see that the memory goes from 9.6g to ... 9.6g, so the GC did
not release any memory at all.

I have tried to modify all the settings of the garbage collector that I
know of, but nothing seems to work:

  • I tried to use the G1 GC
  • I tried to use ParNewGC and ConcMarkSweepGC with the option
    CMSInitiatingOccupancyOnly
  • I changed the CMSInitiatingOccupancyFraction from 75 to 50 to 60

My configuration is:

  • I have three indexes, and the main one is 200GB large
  • The indexes have 5 shards and 1 replica
  • The 2 nodes are 2 m1.xlarge EC2 instances (4 CPU, 15GB Memory, 420GB hard
    drive each)
  • I use the shared S3 gateway
  • The Java version I use is 1.7.0_51 (I was on 1.7.0_05 before, and I
    updated it to see if the problem could come from that)
  • The nodes are on Ubuntu 12.04
  • The heap size given to ES is now 10G (I tried with 11G and 8GB, but same
    problem).
    I also tried to launch the cluster on a huge single node (hi1.4xlarge : 16
    CPU, 50GB Memory - with 32GB heap size for ES, 1024GB SSD drive), but same
    problem.

The ES options I changed are:

  • indices.fielddata.cache.size: 1G (that was for testing, but with or
    without the option, the problem still occurs)
  • bootstrap.mlockall: true

I also changed the threadpool options (because I had some messages telling
me that the queue was full) to:
threadpool:
search:
type: fixed
size: 10
queue_size: -1
But like for the cache size of the field data, with or without this option,
I still have the GC error.

And I also changed the default ports.
But all those things did not change anything, my GC still does not free
memory.

So, what can I do to solve this problem?

Here is a screenshot I got with bigdesk:

https://lh4.googleusercontent.com/-HZ_uAPuJzF4/UuI_E_kuB6I/AAAAAAAAAS8/5uyf9mOaopk/s1600/Capture+d’écran+2014-01-24+à+10.56.44.png

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4923dac2-d439-424c-9686-45ddd0c53163%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.