.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr)
> > [https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fc9a2de4-c409-427f-828f-c4ee93b8c404%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc9a2de4-c409-427f-828f-c4ee93b8c404%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@fred](https://discuss.elastic.co/u/fred)
#### Post date: [October 10, 2014, 12:48pm UTC](https://discuss.elastic.co/t/es-eating-all-memory-despite-jvm-startup-configuration/20155/5 "2014-10-10T12:48:59Z")
Hello,
I’ve a bit more information about that. My ES process creates thousands of threads and doesn’t seem either to close them nor even use most of them, until it eats all the memory and the process crashes (around 25k active threads).
Here’s a dump of the thread\_pool state 5 minutes after launching the node (with almost no trafic). Is there a was to know what ES is trying to do with these threads?
"thread\_pool" : { "generic" : { "threads" : 496, "queue" : 0, "active" : 496, "rejected" : 0, "largest" : 496, "completed" : 108 }, "index" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "get" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "snapshot" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "merge" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "suggest" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "bulk" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "optimize" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "warmer" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "flush" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "search" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "percolate" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 }, "management" : { "threads" : 2, "queue" : 0, "active" : 1, "rejected" : 0, "largest" : 2, "completed" : 61 }, "refresh" : { "threads" : 0, "queue" : 0, "active" : 0, "rejected" : 0, "largest" : 0, "completed" : 0 } },
--
Frédéric de Villamil
Le jeudi 9 octobre 2014 à 09:48, Frédéric de Villamil a écrit :
> Hello,
>
> upgrading is not yet planned.
>
> The cluster is less than 100GB, and this node is a master one, no data, only queries and very little trafic.
>
> My concern is seeing the process taking 6GB RAM despite the JVM being capped at 512M whithout ever triggering the garbage collector. Also, the memory graph is perfectly linear
>
> On Thursday, 9 October 2014 09:27:55 UTC+2, Mark Walkom wrote:
>
> > How much data is in your cluster?
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) (javascript:)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) ([http://www.campaignmonitor.com](http://www.campaignmonitor.com))
> > On 9 October 2014 18:25, David Pilato \<[da...@pilato.fr](mailto:da...@pilato.fr) (javascript:)\> wrote:
> >
> > > Any chance you could try elasticsearch 1.3.4?
> > >
> > > David
> > >
> > > Le 9 oct. 2014 à 07:51, Frédéric de Villamil \<[fr...@botify.com](mailto:fr...@botify.com) (javascript:)\> a écrit :
> > >
> > > > Hello,
> > > >
> > > > I'v been facing a problem on one of my ES nodes for a few days I can't explain myself. The machine was recently rebooted and I seem to have lost something.
> > > >
> > > > Symptoms: the amount of used memory grows until the kernel triggers OOM Killer and the garbage collector is never triggered (logged in graphite using collectd). ES process takes much more RAM that the JVM is allowed to in the init script. Every other machine have had a flat memory consumption for months.
> > > >
> > > > Here, the configuration, node state and startup command line etc.
> > > >
> > > > Thank you very much.
> > > >
> > > > java -version
> > > > java version "1.7.0\_51"
> > > > Java(TM) SE Runtime Environment (build 1.7.0\_51-b13)
> > > > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
> > > >
> > > > Xms and Xmx are low because I want to see the GC trigger without waiting for ages.
> > > >
> > > > /usr/lib/jvm/java-7-oracle/bin/java -Xms512m -Xmx512m -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.0.1.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
> > > >
> > > > Configuration
> > > >
> > > > bootstrap:
> > > > mlockall: false
> > > > cloud:
> > > > aws:
> > > > access\_key: something
> > > > region: us-east-1
> > > > secret\_key: something
> > > > cluster:
> > > > name: robots
> > > > discovery:
> > > > ec2:
> > > > ping\_timeout: 360
> > > > tag:
> > > > Env: production
> > > > type: ec2
> > > > zen:
> > > > minimum\_master\_nodes: 1
> > > > gateway:
> > > > expected\_nodes: 4
> > > > recover\_after\_nodes: 4
> > > > recover\_after\_time: 5m
> > > > http:
> > > > max\_content\_length: 100mb
> > > > index:
> > > > query:
> > > > bool:
> > > > max\_clause\_count: 1000000
> > > > refresh\_interval: 300
> > > > store:
> > > > type: mmapfs
> > > > indices:
> > > > fielddata:
> > > > cache:
> > > > expire: 10m
> > > > size: 30%
> > > > memory:
> > > > index\_buffer\_size: 10%
> > > > network:
> > > > host: 0.0.0.0
> > > > node:
> > > > data: false
> > > > master: true
> > > > name: something
> > > > path:
> > > > data: /mnt/elasticsearch
> > > > logs: /var/log/elasticsearch
> > > >
> > > > Machine recently rebooted because we lost it.
> > > >
> > > > ```
> > > > "blmnqXmzRBKef_3rTyKxgw" : {
> > > > "name" : "something",
> > > > "transport_address" : "inet[/something:9300]",
> > > > "host" : "something",
> > > > "ip" : "something",
> > > > "version" : "1.0.1",
> > > > "build" : "5c03844",
> > > > "http_address" : "inet[/10.146.197.134:9200 (http://10.146.197.134:9200)]",
> > > > "thrift_address" : "/10.146.197.134:9500 (http://10.146.197.134:9500)",
> > > > "attributes" : {
> > > > "data" : "false",
> > > > "master" : "true"
> > > > },
> > > > "settings" : {
> > > > "index" : {
> > > > "store" : {
> > > > "type" : "mmapfs"
> > > > },
> > > > "query" : {
> > > > "bool" : {
> > > > "max_clause_count" : "1000000"
> > > > }
> > > > },
> > > > "refresh_interval" : "300"
> > > > },
> > > > "bootstrap" : {
> > > > "mlockall" : "false"
> > > > },
> > > > "gateway" : {
> > > > "recover_after_time" : "5m",
> > > > "expected_nodes" : "4",
> > > > "recover_after_nodes" : "4"
> > > > },
> > > > "pidfile" : "/var/run/elasticsearch.pid",
> > > > "network" : {
> > > > "host" : "0.0.0.0"
> > > > },
> > > > "node" : {
> > > > "data" : "false",
> > > > "master" : "true",
> > > > "name" : "i-ce8036ed-query.ec2.internal"
> > > > },
> > > > "http" : {
> > > > "max_content_length" : "100mb"
> > > > },
> > > > "name" : "i-ce8036ed-query.ec2.internal",
> > > > "path" : {
> > > > "data" : "/mnt/elasticsearch",
> > > > "work" : "/tmp/elasticsearch",
> > > > "home" : "/usr/share/elasticsearch",
> > > > "conf" : "/etc/elasticsearch",
> > > > "logs" : "/var/log/elasticsearch"
> > > > },
> > > > "cloud" : {
> > > > "aws" : {
> > > > "region" : "us-east-1"
> > > > }
> > > > },
> > > > "config" : "/etc/elasticsearch/elasticsearch.yml",
> > > > "cluster" : {
> > > > "name" : "robots"
> > > > },
> > > > "indices" : {
> > > > "fielddata" : {
> > > > "cache" : { "expire" : "10m",
> > > > "size" : "30%"
> > > > }
> > > > },
> > > > "memory" : {
> > > > "index_buffer_size" : "10%"
> > > > }
> > > > },
> > > > "discovery" : {
> > > > "type" : "ec2",
> > > > "zen" : {
> > > > "minimum_master_nodes" : "1"
> > > > },
> > > > "ec2" : {
> > > > "ping_timeout" : "360",
> > > > "tag" : {
> > > > "Env" : "production"
> > > > }
> > > > }
> > > > },
> > > > "foreground" : "yes"
> > > > },
> > > > "os" : {
> > > > "refresh_interval" : 1000,
> > > > "available_processors" : 2,
> > > > "cpu" : {
> > > > "vendor" : "Intel",
> > > > "model" : "Xeon",
> > > > "mhz" : 2500,
> > > > "total_cores" : 2,
> > > > "total_sockets" : 2,
> > > > "cores_per_socket" : 32,
> > > > "cache_size_in_bytes" : 25600
> > > > },
> > > > "mem" : {
> > > > "total_in_bytes" : 7812546560
> > > > },
> > > > "swap" : {
> > > > "total_in_bytes" : 0
> > > > }
> > > > },
> > > > "process" : {
> > > > "refresh_interval" : 1000,
> > > > "id" : 1638,
> > > > "max_file_descriptors" : 65535,
> > > > "mlockall" : false
> > > > },
> > > > "jvm" : {
> > > > "pid" : 1638,
> > > > "version" : "1.7.0_51",
> > > > "vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
> > > > "vm_version" : "24.51-b03",
> > > > "vm_vendor" : "Oracle Corporation",
> > > > "start_time" : 1412832661002,
> > > > "mem" : {
> > > > "heap_init_in_bytes" : 536870912,
> > > > "heap_max_in_bytes" : 519438336,
> > > > "non_heap_init_in_bytes" : 24313856,
> > > > "non_heap_max_in_bytes" : 136314880,
> > > > "direct_max_in_bytes" : 519438336
> > > > },
> > > > },
> > > > "gc_collectors" : ["ParNew", "ConcurrentMarkSweep"],
> > > > "memory_pools" : ["Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen"]
> > > > },
> > > > "thread_pool" : {
> > > > "generic" : {
> > > > "type" : "cached",
> > > > "keep_alive" : "30s"
> > > > },
> > > > "index" : {
> > > > "type" : "fixed",
> > > > "min" : 2,
> > > > "max" : 2,
> > > > "queue_size" : "200"
> > > > },
> > > > "get" : {
> > > > "type" : "fixed",
> > > > "min" : 2,
> > > > "max" : 2,
> > > > "queue_size" : "1k"
> > > > },
> > > > "snapshot" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 1,
> > > > "keep_alive" : "5m"
> > > > }, "merge" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 1,
> > > > "keep_alive" : "5m"
> > > > },
> > > > "suggest" : {
> > > > "type" : "fixed",
> > > > "min" : 2,
> > > > "max" : 2,
> > > > "queue_size" : "1k"
> > > > },
> > > > "bulk" : {
> > > > "type" : "fixed",
> > > > "min" : 2,
> > > > "max" : 2,
> > > > "queue_size" : "50"
> > > > },
> > > > "optimize" : {
> > > > "type" : "fixed",
> > > > "min" : 1,
> > > > "max" : 1
> > > > },
> > > > "warmer" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 1,
> > > > "keep_alive" : "5m"
> > > > }, "flush" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 1,
> > > > "keep_alive" : "5m"
> > > > },
> > > > "search" : {
> > > > "type" : "fixed",
> > > > "min" : 6,
> > > > "max" : 6,
> > > > "queue_size" : "1k"
> > > > },
> > > > "percolate" : {
> > > > "type" : "fixed",
> > > > "min" : 2,
> > > > "max" : 2,
> > > > "queue_size" : "1k"
> > > > },
> > > > "management" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 5,
> > > > "keep_alive" : "5m"
> > > > },
> > > > "refresh" : {
> > > > "type" : "scaling",
> > > > "min" : 1,
> > > > "max" : 1,
> > > > "keep_alive" : "5m"
> > > > }
> > > > },
> > > > "network" : {
> > > > "refresh_interval" : 5000,
> > > > "primary_interface" : {
> > > > "address" : "something",
> > > > "name" : "eth0",
> > > > "mac_address" : "22:00:0B:2F:90:D8"
> > > > }
> > > > },
> > > > "transport" : {
> > > > "bound_address" : "inet[/0:0:0:0:0:0:0:0:9300]",
> > > > "publish_address" : "inet[/10.146.197.134:9300 (http://10.146.197.134:9300)]"
> > > > },
> > > > "http" : {
> > > > "bound_address" : "inet[/0:0:0:0:0:0:0:0:9200]",
> > > > "publish_address" : "inet[/something:9200]",
> > > > "max_content_length_in_bytes" : 104857600
> > > > },
> > > > "plugins" : [ {
> > > > "name" : "cloud-aws",
> > > > "version" : "NA",
> > > > "description" : "Cloud AWS Plugin",
> > > > "jvm" : true,
> > > > "site" : false
> > > > }, {
> > > > "name" : "transport-thrift",
> > > > "version" : "2.0.0",
> > > > "description" : "Exports elasticsearch REST APIs over thrift",
> > > > "jvm" : true,
> > > > "site" : false
> > > > }, {
> > > > "name" : "head",
> > > > "version" : "NA",
> > > > "description" : "No description found.",
> > > > "url" : "/_plugin/head/",
> > > > "jvm" : false,
> > > > "site" : true
> > > > } ]
> > > > },
> > > >
> > > > ```
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) (javascript:).
> > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/070aef63-0a1f-4bd0-820b-96b1b79cf742%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/070aef63-0a1f-4bd0-820b-96b1b79cf742%40googlegroups.com) ([https://groups.google.com/d/msgid/elasticsearch/070aef63-0a1f-4bd0-820b-96b1b79cf742%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/070aef63-0a1f-4bd0-820b-96b1b79cf742%40googlegroups.com?utm_medium=email&utm_source=footer)).
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > --
> > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) (javascript:).
> > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr) ([https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/60AB40E4-6BEF-44CD-BA4F-095B0090AB34%40pilato.fr?utm_medium=email&utm_source=footer)).
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/M6tUbchMYko/unsubscribe](https://groups.google.com/d/topic/elasticsearch/M6tUbchMYko/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) ([mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fc9a2de4-c409-427f-828f-c4ee93b8c404%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fc9a2de4-c409-427f-828f-c4ee93b8c404%40googlegroups.com) ([https://groups.google.com/d/msgid/elasticsearch/fc9a2de4-c409-427f-828f-c4ee93b8c404%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/fc9a2de4-c409-427f-828f-c4ee93b8c404%40googlegroups.com?utm_medium=email&utm_source=footer)).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3A0DF1E7BB7F48C6878882E56BEC1648%40botify.com](https://groups.google.com/d/msgid/elasticsearch/3A0DF1E7BB7F48C6878882E56BEC1648%40botify.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@sureshadapa](https://discuss.elastic.co/u/sureshadapa)
#### Post date: [November 19, 2015, 8:40am UTC](https://discuss.elastic.co/t/es-eating-all-memory-despite-jvm-startup-configuration/20155/6 "2015-11-19T08:40:16Z")
Hi,
I have Linux - el6.x86\_64, 64 bit machine on which ES-1.71. is running with 2GB, and my JVM setting are as below.
/usr/bin/java -Xms2g -Xmx2g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/sw/elasticsearch-1.7.1 -cp :/sw/elasticsearch-1.7.1/lib/elasticsearch-1.7.1.jar:/sw/elasticsearch-1.7.1/lib/_:/sw/elasticsearch-1.7.1/lib/sigar/_ org.elasticsearch.bootstrap.Elasticsearch
When "top" is ran on this machine which has 8GB RAM almost 7.5GB is used.I feel that ES is not releasing the Memory. Do you recommend changing CMSInitiatingOccupancyFraction to 50?
And any other JVM tuning parameters for ES and debugging options.
I notice the below on Threads.
Live:78
Daemon:75
Live Peak:86
Total Started:13,428
Thanks
SureshA
---
### Author:  [@sureshadapa](https://discuss.elastic.co/u/sureshadapa)
#### Post date: [November 19, 2015, 8:50am UTC](https://discuss.elastic.co/t/es-eating-all-memory-despite-jvm-startup-configuration/20155/7 "2015-11-19T08:50:23Z")
i am using
openjdk version "1.8.0\_31"
OpenJDK Runtime Environment (build 1.8.0\_31-b13)
OpenJDK 64-Bit Server VM (build 25.31-b07, mixed mode)
w.r.t "disable the Java 7 default G1 collector if you run Elasticsearch on Java 7 later than 7u4" at [http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html](http://jprante.github.io/2012/11/28/Elasticsearch-Java-Virtual-Machine-settings-explained.html)
My java process has -XX:+DisableExplicitGC
---
### Author:  [@jpountz](https://discuss.elastic.co/u/jpountz)
#### Post date: [November 19, 2015, 2:46pm UTC](https://discuss.elastic.co/t/es-eating-all-memory-despite-jvm-startup-configuration/20155/8 "2015-11-19T14:46:22Z")
This is because elasticsearch uses mmap in order to read files. It does not actually use memory (your machine probably has a lot of free physical memory) however it uses some virtual address space (this is how mmap works). Thers is really no problem, when monitoring memory usage, just make sure to monitor how much physical memory is used, not virtual address space.
See [http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html](http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html) for more detailed information about this.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:37pm UTC](https://discuss.elastic.co/t/es-eating-all-memory-despite-jvm-startup-configuration/20155/9 "2017-07-05T23:37:14Z")