# Es got blocked **URL:** \ **Category:** Elasticsearch\ **Created:** [December 12, 2011, 4:00pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129 "2011-12-12T16:00:55Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Sisu\_Alexandru](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@Sisu\_Alexandru](https://discuss.elastic.co/u/Sisu_Alexandru)\ **Post date:** [December 12, 2011, 4:00pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/1 "2011-12-12T16:00:55Z")
Hello all, Our es suddenly got blocked for 15 minutes. That means: it suddenly stopped handling search requests and also status requests:curl -XGET [http://localhost:9200/\_status](http://localhost:9200/_status). ES version: 0.18.5 The size of the index is arround 200G. One ES client. 20 shards. All on one single machine. No mirrors. OS: centos 5. Here is the gist: [https://gist.github.com/1467971](https://gist.github.com/1467971) This info was obtained by jstack. (jstack -F 27029 (pid of es)). It looks like all the threads got blocked (all 132 )?! The size of the jstack output is much more bigger I've copy pasted only some parts of it. Any ideas? Tnx in advance, Alex ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [December 12, 2011, 4:40pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/2 "2011-12-12T16:40:11Z")
Garbage Collection? How much memory are you giving each JVM? If it's a large amount and you haven't tuned your GC options on the JVM, this is a likely cause. I don't suppose you had something monitoring JMX over that time period. If you did you'd be able to see if this was the issue if you notice the Heap Space Used dropping off. Paul. On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > Hello all, > > Our es suddenly got blocked for 15 minutes. That means: it suddenly > stopped handling search requests and also status requests:curl -XGET > [http://localhost:9200/\_status](http://localhost:9200/_status). > > ES version: 0.18.5 > The size of the index is arround 200G. > One ES client. > 20 shards. All on one single machine. > No mirrors. > OS: centos 5. > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It looks > like all the threads got blocked (all 132 )?! > > The size of the jstack output is much more bigger I've copy pasted only > some parts of it. > > Any ideas? > > Tnx in advance, > > Alex ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Sisu\_Alexandru](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@Sisu\_Alexandru](https://discuss.elastic.co/u/Sisu_Alexandru)\ **Post date:** [December 12, 2011, 9:12pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/3 "2011-12-12T21:12:42Z")
Well I started up Elasticsearch with Xms and Xmx set to 100G. That should've been enough. Monitoring though JMX sounds a good ideea. but how can you configure jmx options in es? The only documentation that I found was here [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its not very clear to me what should I do. Tnx, Alex On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > Garbage Collection? How much memory are you giving each JVM? If it's a > large amount and you haven't tuned your GC options on the JVM, this is a > likely cause. > > I don't suppose you had something monitoring JMX over that time period. If > you did you'd be able to see if this was the issue if you notice the Heap > Space Used dropping off. > > Paul. > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > Hello all, > > > > Our es suddenly got blocked for 15 minutes. That means: it suddenly > > stopped handling search requests and also status requests:curl -XGET > > [http://localhost:9200/\_status](http://localhost:9200/_status). > > > > ES version: 0.18.5 > > The size of the index is arround 200G. > > One ES client. > > 20 shards. All on one single machine. > > No mirrors. > > OS: centos 5. > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It looks > > like all the threads got blocked (all 132 )?! > > > > The size of the jstack output is much more bigger I've copy pasted only > > some parts of it. > > > > Any ideas? > > > > Tnx in advance, > > > > Alex > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 12, 2011, 10:43pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/4 "2011-12-12T22:43:56Z")
You configured ES with 100g? Do you have a machine with a 100gb of memory? How much memory does your machine has? I am surprise that it even started... (swap might be really big). On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > Well I started up Elasticsearch with Xms and Xmx set to 100G. > That should've been enough. > > Monitoring though JMX sounds a good ideea. but how can you configure jmx > options in es? > The only documentation that I found was here > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its not > very clear to me what should I do. > > Tnx, > > Alex > > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > Garbage Collection? How much memory are you giving each JVM? If it's a > > large amount and you haven't tuned your GC options on the JVM, this is a > > likely cause. > > > > I don't suppose you had something monitoring JMX over that time period. > > If you did you'd be able to see if this was the issue if you notice the > > Heap Space Used dropping off. > > > > Paul. > > > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > Hello all, > > > > > > Our es suddenly got blocked for 15 minutes. That means: it suddenly > > > stopped handling search requests and also status requests:curl -XGET > > > [http://localhost:9200/\_status](http://localhost:9200/_status). > > > > > > ES version: 0.18.5 > > > The size of the index is arround 200G. > > > One ES client. > > > 20 shards. All on one single machine. > > > No mirrors. > > > OS: centos 5. > > > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It > > > looks like all the threads got blocked (all 132 )?! > > > > > > The size of the jstack output is much more bigger I've copy pasted only > > > some parts of it. > > > > > > Any ideas? > > > > > > Tnx in advance, > > > > > > Alex > > > > ## -- > > > > Paul Loy > > [paul@keteracel.com](mailto:paul@keteracel.com) > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Sisu\_Alexandru](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@Sisu\_Alexandru](https://discuss.elastic.co/u/Sisu_Alexandru)\ **Post date:** [December 13, 2011, 8:54am UTC](https://discuss.elastic.co/t/es-got-blocked/6129/5 "2011-12-13T08:54:42Z")
Yes, the machine has 128 GB , 48 cores. And till now we didn't have no problems. Anyhow, yesterday I restarted the machine, and everything got back to normal. Still I'm interesting in how to configure the jmx for es. Any tips? 🙂 On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > You configured ES with 100g? Do you have a machine with a 100gb of memory? > How much memory does your machine has? I am surprise that it even > started... (swap might be really big). > > On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > Well I started up Elasticsearch with Xms and Xmx set to 100G. > > That should've been enough. > > > > Monitoring though JMX sounds a good ideea. but how can you configure jmx > > options in es? > > The only documentation that I found was here > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its > > not very clear to me what should I do. > > > > Tnx, > > > > Alex > > > > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > Garbage Collection? How much memory are you giving each JVM? If it's a > > > large amount and you haven't tuned your GC options on the JVM, this is a > > > likely cause. > > > > > > I don't suppose you had something monitoring JMX over that time period. > > > If you did you'd be able to see if this was the issue if you notice the > > > Heap Space Used dropping off. > > > > > > Paul. > > > > > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > > > Hello all, > > > > > > > > Our es suddenly got blocked for 15 minutes. That means: it suddenly > > > > stopped handling search requests and also status requests:curl -XGET > > > > [http://localhost:9200/\_status](http://localhost:9200/_status). > > > > > > > > ES version: 0.18.5 > > > > The size of the index is arround 200G. > > > > One ES client. > > > > 20 shards. All on one single machine. > > > > No mirrors. > > > > OS: centos 5. > > > > > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > > > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It > > > > looks like all the threads got blocked (all 132 )?! > > > > > > > > The size of the jstack output is much more bigger I've copy pasted > > > > only some parts of it. > > > > > > > > Any ideas? > > > > > > > > Tnx in advance, > > > > > > > > Alex > > > > > > ## -- > > > > > > Paul Loy > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [December 13, 2011, 10:17am UTC](https://discuss.elastic.co/t/es-got-blocked/6129/6 "2011-12-13T10:17:33Z")
That's some machine! Yeah, with stutters like this it's very useful to know what's going on with your heap (and other resources). You can watch the heap via JConsole or use some monitoring tool like traverse to fire emails when the heap gets too big or simply be able to view historical graphs of all your JMX exposed variables. To enable JMX it looks like you need jmx.create\_connector: true in your elasticsearch.yml. VisualVM also has an awesome Visual GC plugin that lets you see which of the various sections of your Heap are filling up. Cheers, Paul. On Tue, Dec 13, 2011 at 12:54 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > Yes, the machine has 128 GB , 48 cores. And till now we didn't have no > problems. > Anyhow, yesterday I restarted the machine, and everything got back to > normal. > Still I'm interesting in how to configure the jmx for es. Any tips? 🙂 > > On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > You configured ES with 100g? Do you have a machine with a 100gb of > > memory? How much memory does your machine has? I am surprise that it even > > started... (swap might be really big). > > > > On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > Well I started up Elasticsearch with Xms and Xmx set to 100G. > > > That should've been enough. > > > > > > Monitoring though JMX sounds a good ideea. but how can you configure jmx > > > options in es? > > > The only documentation that I found was here > > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its > > > not very clear to me what should I do. > > > > > > Tnx, > > > > > > Alex > > > > > > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > > > Garbage Collection? How much memory are you giving each JVM? If it's a > > > > large amount and you haven't tuned your GC options on the JVM, this is a > > > > likely cause. > > > > > > > > I don't suppose you had something monitoring JMX over that time period. > > > > If you did you'd be able to see if this was the issue if you notice the > > > > Heap Space Used dropping off. > > > > > > > > Paul. > > > > > > > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > > > > > Hello all, > > > > > > > > > > Our es suddenly got blocked for 15 minutes. That means: it suddenly > > > > > stopped handling search requests and also status requests:curl -XGET > > > > > [http://localhost:9200/\_status](http://localhost:9200/_status). > > > > > > > > > > ES version: 0.18.5 > > > > > The size of the index is arround 200G. > > > > > One ES client. > > > > > 20 shards. All on one single machine. > > > > > No mirrors. > > > > > OS: centos 5. > > > > > > > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > > > > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It > > > > > looks like all the threads got blocked (all 132 )?! > > > > > > > > > > The size of the jstack output is much more bigger I've copy pasted > > > > > only some parts of it. > > > > > > > > > > Any ideas? > > > > > > > > > > Tnx in advance, > > > > > > > > > > Alex > > > > > > > > ## -- > > > > > > > > Paul Loy > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 13, 2011, 10:23pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/7 "2011-12-13T22:23:54Z")
Also, I would add that you make sure to enable mlockall in the configuration to make sure the OS will not swap the elasticsearch process. I never ran ES with 100gb of memory, whats your typical memory usage? (node stats can give you a lot of information, also on the jvm level). On Tue, Dec 13, 2011 at 12:17 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > That's some machine! > > Yeah, with stutters like this it's very useful to know what's going on > with your heap (and other resources). You can watch the heap via JConsole > or use some monitoring tool like traverse to fire emails when the heap gets > too big or simply be able to view historical graphs of all your JMX exposed > variables. > > To enable JMX it looks like you need jmx.create\_connector: true in your > elasticsearch.yml. > > VisualVM also has an awesome Visual GC plugin that lets you see which of > the various sections of your Heap are filling up. > > Cheers, > > Paul. > > On Tue, Dec 13, 2011 at 12:54 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > Yes, the machine has 128 GB , 48 cores. And till now we didn't have no > > problems. > > Anyhow, yesterday I restarted the machine, and everything got back to > > normal. > > Still I'm interesting in how to configure the jmx for es. Any tips? 🙂 > > > > On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > You configured ES with 100g? Do you have a machine with a 100gb of > > > memory? How much memory does your machine has? I am surprise that it even > > > started... (swap might be really big). > > > > > > On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > > > Well I started up Elasticsearch with Xms and Xmx set to 100G. > > > > That should've been enough. > > > > > > > > Monitoring though JMX sounds a good ideea. but how can you configure > > > > jmx options in es? > > > > The only documentation that I found was here > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its > > > > not very clear to me what should I do. > > > > > > > > Tnx, > > > > > > > > Alex > > > > > > > > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > > > > > Garbage Collection? How much memory are you giving each JVM? If it's a > > > > > large amount and you haven't tuned your GC options on the JVM, this is a > > > > > likely cause. > > > > > > > > > > I don't suppose you had something monitoring JMX over that time > > > > > period. If you did you'd be able to see if this was the issue if you notice > > > > > the Heap Space Used dropping off. > > > > > > > > > > Paul. > > > > > > > > > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > Our es suddenly got blocked for 15 minutes. That means: it suddenly > > > > > > stopped handling search requests and also status requests:curl -XGET > > > > > > [http://localhost:9200/\_status](http://localhost:9200/_status). > > > > > > > > > > > > ES version: 0.18.5 > > > > > > The size of the index is arround 200G. > > > > > > One ES client. > > > > > > 20 shards. All on one single machine. > > > > > > No mirrors. > > > > > > OS: centos 5. > > > > > > > > > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > > > > > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It > > > > > > looks like all the threads got blocked (all 132 )?! > > > > > > > > > > > > The size of the jstack output is much more bigger I've copy pasted > > > > > > only some parts of it. > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > Tnx in advance, > > > > > > > > > > > > Alex > > > > > > > > > > ## -- > > > > > > > > > > Paul Loy > > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Sisu\_Alexandru](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@Sisu\_Alexandru](https://discuss.elastic.co/u/Sisu_Alexandru)\ **Post date:** [December 19, 2011, 2:11pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/8 "2011-12-19T14:11:30Z")
Hello all, Following your suggestion I've tried: - running with bootstrap.mlockall set to True. - i've enabled the JMX monitoring. The es continues to hangs. I can reproduce the problem again and again, by executing the following query: { "facets": { "term\_count": { "global": true, "terms": { "field": "body", "size": 100 } } }, "size": 0} (I'm trying to retrieve the most common 100 terms for this field). To my surprise, I've discovered in the es folder, a set of huge files:\* java\_pid\__xxx_.hprof\*. Generated each time my es got\* 'blocked'.\* Now, this files are generated when the process runs out of memory. I'm running with 100G of heap memory allocated, and I expect that for large memory consuming operations the memory to be swapped. Anyhow, it seems that on my machine, each time I'm running the above query, I'm killing it. Other informations: - the jhat heap histogram can be found here: [jhat\_es\_hprof\_stat · GitHub](https://gist.github.com/1497379) - through _JConsole_, each time I'm executing this query, I can see how the heap memory increases !very fast! it seems that 100G are consumed in a few seconds. - regarding the node stats and cluster stats es provides: I cannot access them after es dies, but here is the info of es, right after restart when everything is okey: os: { refresh\_interval: 1000 cpu: { vendor: AMD model: Opteron mhz: 1900 total\_cores: 48 total\_sockets: 4 cores\_per\_socket: 12 cache\_size: 512b cache\_size\_in\_bytes: 512 } mem: { total: 126gb total\_in\_bytes: 135321870336 } swap: { total: 64gb total\_in\_bytes: 68803354624 } } process: { refresh\_interval: 1000 id: 15926 max\_file\_descriptors: 128000 } jvm: { pid: 15926 version: 1.6.0\_27 vm\_name: Java HotSpot(TM) 64-Bit Server VM vm\_version: 20.2-b06 vm\_vendor: Sun Microsystems Inc. start\_time: 1324275713418 mem: { heap\_init: 100gb heap\_init\_in\_bytes: 107374182400 heap\_max: 99.9gb heap\_max\_in\_bytes: 107302223872 non\_heap\_init: 23.1mb non\_heap\_init\_in\_bytes: 24313856 non\_heap\_max: 130mb non\_heap\_max\_in\_bytes: 136314880 } } Any suggestions? Tnx, Alex On Tue, Dec 13, 2011 at 11:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Also, I would add that you make sure to enable mlockall in the > configuration to make sure the OS will not swap the elasticsearch process. > I never ran ES with 100gb of memory, whats your typical memory usage? (node > stats can give you a lot of information, also on the jvm level). > > On Tue, Dec 13, 2011 at 12:17 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > That's some machine! > > > > Yeah, with stutters like this it's very useful to know what's going on > > with your heap (and other resources). You can watch the heap via JConsole > > or use some monitoring tool like traverse to fire emails when the heap gets > > too big or simply be able to view historical graphs of all your JMX exposed > > variables. > > > > To enable JMX it looks like you need jmx.create\_connector: true in your > > elasticsearch.yml. > > > > VisualVM also has an awesome Visual GC plugin that lets you see which of > > the various sections of your Heap are filling up. > > > > Cheers, > > > > Paul. > > > > On Tue, Dec 13, 2011 at 12:54 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > Yes, the machine has 128 GB , 48 cores. And till now we didn't have no > > > problems. > > > Anyhow, yesterday I restarted the machine, and everything got back to > > > normal. > > > Still I'm interesting in how to configure the jmx for es. Any tips? 🙂 > > > > > > On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > You configured ES with 100g? Do you have a machine with a 100gb of > > > > memory? How much memory does your machine has? I am surprise that it even > > > > started... (swap might be really big). > > > > > > > > On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > > > > > Well I started up Elasticsearch with Xms and Xmx set to 100G. > > > > > That should've been enough. > > > > > > > > > > Monitoring though JMX sounds a good ideea. but how can you configure > > > > > jmx options in es? > > > > > The only documentation that I found was here > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its > > > > > not very clear to me what should I do. > > > > > > > > > > Tnx, > > > > > > > > > > Alex > > > > > > > > > > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > > > > > > > Garbage Collection? How much memory are you giving each JVM? If it's > > > > > > a large amount and you haven't tuned your GC options on the JVM, this is a > > > > > > likely cause. > > > > > > > > > > > > I don't suppose you had something monitoring JMX over that time > > > > > > period. If you did you'd be able to see if this was the issue if you notice > > > > > > the Heap Space Used dropping off. > > > > > > > > > > > > Paul. > > > > > > > > > > > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru \<[sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > Our es suddenly got blocked for 15 minutes. That means: it > > > > > > > suddenly stopped handling search requests and also status requests:curl > > > > > > > -XGET [http://localhost:9200/\_status](http://localhost:9200/_status). > > > > > > > > > > > > > > ES version: 0.18.5 > > > > > > > The size of the index is arround 200G. > > > > > > > One ES client. > > > > > > > 20 shards. All on one single machine. > > > > > > > No mirrors. > > > > > > > OS: centos 5. > > > > > > > > > > > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > > > > > > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It > > > > > > > looks like all the threads got blocked (all 132 )?! > > > > > > > > > > > > > > The size of the jstack output is much more bigger I've copy pasted > > > > > > > only some parts of it. > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > Tnx in advance, > > > > > > > > > > > > > > Alex > > > > > > > > > > > > ## -- > > > > > > > > > > > > Paul Loy > > > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > > > ## -- > > > > Paul Loy > > [paul@keteracel.com](mailto:paul@keteracel.com) > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Aurelien\_2](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@Aurelien\_2](https://discuss.elastic.co/u/Aurelien_2)\ **Post date:** [December 19, 2011, 2:35pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/9 "2011-12-19T14:35:55Z")
Hello. hprof files are generated by - XX:+HeapDumpOnOutOfMemoryError parameter. The files should be roughly same size as heap size when OOM occurs, but could you give us the size of those files? jhat profiles does not show a 100GB used heap, but do you have any OOM error in log files? I don't know if you modified launch scripts, but theses errors should be redirected via stderr. Do you have a huge CPU consommation on 1 or more CPU during the "es blocked" situation? It should be interesting to: - redirect stderr and stdout to a file if it's not done already (with &\> /path/to/file.log at the end of java command file) - activate verbose gc to a specific file - launch visualvm with visualgc plugin - launch recurrent thread dumps (with kill -3 ) within short time period (one thread dump every 10 or 15 seconds) and then reproduce the problem asap to avoid generating to much logs. If you have an OOM, you will wich generation is full with the verbosegc and visualgc, if you have a problem with threads you will see them in thread dumps (I use samurai [http://yusuke.homeip.net/samurai/](http://yusuke.homeip.net/samurai/) to analyze thread dumps), MAT to analyze heap dump/hprof files (but huge hprof should be difficult to read, maybe with jhat). Samurai can read verbosegc files too. In any case, try to use a more recent JVM, please send the complete command line of your ES, and maybe try another JDK (like jrockit with either generation/non generational GC). 100GB is a quite huge memory for actual GC, see [Understanding Java Garbage Collection and What You Can Do about It](http://www.infoq.com/presentations/Understanding-Java-Garbage-Collection) Rgds. ----- Mail original ----- > De: "Sisu Alexandru" [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) > À: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > Envoyé: Lundi 19 Décembre 2011 15:11:30 > Objet: Re: es got blocked > Hello all, > Following your suggestion I've tried: > > - running with bootstrap.mlockall set to True. > - i've enabled the JMX monitoring. > The es continues to hangs. I can reproduce the problem again and > again, by executing the following query: > { "facets": { "term\_count": { "global": true, "terms": { "field": > "body", "size": 100 } } }, "size": 0} > (I'm trying to retrieve the most common 100 terms for this field). > To my surprise, I've discovered in the es folder, a set of huge > files: java\_pid\_ xxx .hprof .. Generated each time my es got > 'blocked'. > Now, this files are generated when the process runs out of memory. > I'm running with 100G of heap memory allocated, and I expect that for > large memory consuming operations the memory to be swapped. > Anyhow, it seems that on my machine, each time I'm running the above > query, I'm killing it. > Other informations: > > - the jhat heap histogram can be found here: > [jhat\_es\_hprof\_stat · GitHub](https://gist.github.com/1497379) > - through JConsole , each time I'm executing this query, I can see > how the heap memory increases !very fast! it seems that 100G are > consumed in a few seconds. > - regarding the node stats and cluster stats es provides: I cannot > access them after es dies, but here is the info of es, right after > restart when everything is okey: > os: { > refresh\_interval: 1000 > cpu: { > vendor: AMD > model: Opteron > mhz: 1900 > total\_cores: 48 > total\_sockets: 4 > cores\_per\_socket: 12 > cache\_size: 512b > cache\_size\_in\_bytes: 512 > } > mem: { > total: 126gb > total\_in\_bytes: 135321870336 > } > swap: { > total: 64gb > total\_in\_bytes: 68803354624 > } > } > process: { > refresh\_interval: 1000 > id: 15926 > max\_file\_descriptors: 128000 > } > jvm: { > pid: 15926 > version: 1.6..0\_27 > vm\_name: Java HotSpot(TM) 64-Bit Server VM > vm\_version: 20.2-b06 > vm\_vendor: Sun Microsystems Inc. > start\_time: 1324275713418 > mem: { > heap\_init: 100gb > heap\_init\_in\_bytes: 107374182400 > heap\_max: 99.9gb > heap\_max\_in\_bytes: 107302223872 > non\_heap\_init: 23.1mb > non\_heap\_init\_in\_bytes: 24313856 > non\_heap\_max: 130mb > non\_heap\_max\_in\_bytes: 136314880 > } > } > Any suggestions? > Tnx, > Alex > On Tue, Dec 13, 2011 at 11:23 PM, Shay Banon \< [kimchy@gmail.com](mailto:kimchy@gmail.com) \> > wrote: > > Also, I would add that you make sure to enable mlockall in the > > configuration to make sure the OS will not swap the elasticsearch > > process. I never ran ES with 100gb of memory, whats your typical > > memory usage? (node stats can give you a lot of information, also > > on > > the jvm level). > > On Tue, Dec 13, 2011 at 12:17 PM, Paul Loy \< [keteracel@gmail.com](mailto:keteracel@gmail.com) \> > > wrote: > > > That's some machine! > > > Yeah, with stutters like this it's very useful to know what's > > > going > > > on with your heap (and other resources). You can watch the heap > > > via > > > JConsole or use some monitoring tool like traverse to fire emails > > > when the heap gets too big or simply be able to view historical > > > graphs of all your JMX exposed variables. > > > To enable JMX it looks like you need jmx.create\_connector: true > > > in > > > your elasticsearch.yml . > > > VisualVM also has an awesome Visual GC plugin that lets you see > > > which > > > of the various sections of your Heap are filling up. > > > Cheers, > > > Paul. > > > On Tue, Dec 13, 2011 at 12:54 AM, Sisu Alexandru \< > > > [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) \> wrote: > > > > Yes, the machine has 128 GB , 48 cores. And till now we didn't > > > > have > > > > no problems. > > > > > Anyhow, yesterday I restarted the machine, and everything got > > > > back > > > > to > > > > normal. > > > > > Still I'm interesting in how to configure the jmx for es. Any > > > > tips? > > > > 🙂 > > > > On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon \< [kimchy@gmail.com](mailto:kimchy@gmail.com) > > > > > > > > > > > > > > > > > wrote: > > > > > You configured ES with 100g? Do you have a machine with a > > > > > 100gb > > > > > of > > > > > memory? How much memory does your machine has? I am surprise > > > > > that > > > > > it > > > > > even started.... (swap might be really big). > > > > > On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru \< > > > > > [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) \> wrote: > > > > > > Well I started up Elasticsearch with Xms and Xmx set to > > > > > > 100G. > > > > > > > That should've been enough. > > > > > > Monitoring though JMX sounds a good ideea. but how can you > > > > > > configure > > > > > > jmx options in es? > > > > > > > The only documentation that I found was here > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) > > > > > > but > > > > > > its not very clear to me what should I do. > > > > > > Tnx, > > > > > > Alex > > > > > > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy \< > > > > > > [keteracel@gmail.com](mailto:keteracel@gmail.com) > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > Garbage Collection? How much memory are you giving each > > > > > > > JVM? > > > > > > > If > > > > > > > it's > > > > > > > a large amount and you haven't tuned your GC options on > > > > > > > the > > > > > > > JVM, > > > > > > > this is a likely cause. > > > > > > > I don't suppose you had something monitoring JMX over > > > > > > > that > > > > > > > time > > > > > > > period. If you did you'd be able to see if this was the > > > > > > > issue > > > > > > > if > > > > > > > you > > > > > > > notice the Heap Space Used dropping off. > > > > > > > Paul. > > > > > > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru \< > > > > > > > [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) \> wrote: > > > > > > > > Hello all, > > > > > > > > Our es suddenly got blocked for 15 minutes.. That > > > > > > > > means: > > > > > > > > it > > > > > > > > suddenly > > > > > > > > stopped handling search requests and also status > > > > > > > > requests:curl > > > > > > > > -XGET > > > > > > > > [http://localhost:9200/\_status](http://localhost:9200/_status) . > > > > > > > > ES version: 0.18.5 > > > > > > > > > The size of the index is arround 200G. > > > > > > > > > One ES client. > > > > > > > > > 20 shards. All on one single machine. > > > > > > > > > No mirrors. > > > > > > > > > OS: centos 5. > > > > > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > > > > > > > > This info was obtained by jstack. (jstack -F 27029 (pid > > > > > > > > of > > > > > > > > es)). > > > > > > > > It > > > > > > > > looks like all the threads got blocked (all 132 )?! > > > > > > > > The size of the jstack output is much more bigger I've > > > > > > > > copy > > > > > > > > pasted > > > > > > > > only some parts of it. > > > > > > > > Any ideas? > > > > > > > > Tnx in advance, > > > > > > > > Alex > > > > > > > -- > > > > > > > > * * * > > > > > > > > Paul Loy > > > > > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > > -- > > > > * * * > > > > Paul Loy > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Aurelien\_2](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@Aurelien\_2](https://discuss.elastic.co/u/Aurelien_2)\ **Post date:** [December 19, 2011, 2:37pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/10 "2011-12-19T14:37:53Z")
Hello. hprof files are generated by -XX:+HeapDumpOnOutOfMemoryError parameter. The files should be roughly same size as heap size when OOM occurs, but could you give us the size of those files? jhat profiles does not show a 100GB used heap, but do you have any OOM error in log files? I don't know if you modified launch scripts, but theses errors should be redirected via stderr. Do you have a huge CPU consommation on 1 or more CPU during the "es blocked" situation? It should be interesting to: - redirect stderr and stdout to a file if it's not done already (with &\> /path/to/file.log at the end of java command file) - activate verbose gc to a specific file - launch visualvm with visualgc plugin - launch recurrent thread dumps (with kill -3 ) within short time period (one thread dump every 10 or 15 seconds) and then reproduce the problem asap to avoid generating to much logs. If you have an OOM, you will wich generation is full with the verbosegc and visualgc, if you have a problem with threads you will see them in thread dumps (I use samurai [http://yusuke.homeip.net/samurai/](http://yusuke.homeip.net/samurai/) to analyze thread dumps), MAT to analyze heap dump/hprof files (but huge hprof should be difficult to read, maybe with jhat). Samurai can read verbosegc files too. In any case, try to use a more recent JVM, please send the complete command line of your ES, and maybe try another JDK (like jrockit with either generation/non generational GC). 100GB is a quite huge memory for actual GC, see > **[Understanding Java Garbage Collection and What You Can Do about It](https://www.infoq.com/presentations/Understanding-Java-Garbage-Collection)** > > Gil Tene explains the workings of a garbage collector: terminology, metrics, fundamentals, key mechanisms, classification of current GCs, the “Application Memory Wall” problem, and details Azul C4 GC. Rgds. ``` De: "Sisu Alexandru" À: elasticsearch@googlegroups.com Envoyé: Lundi 19 Décembre 2011 15:11:30 Objet: Re: es got blocked Hello all, Following your suggestion I've tried: - running with bootstrap.mlockall set to True. - i've enabled the JMX monitoring. The es continues to hangs. I can reproduce the problem again and ``` again, by executing the following query: { "facets": { "term\_count": { "global": true, "terms": { "field": "body", "size": 100 } } }, "size": 0} (I'm trying to retrieve the most common 100 terms for this field). To my surprise, I've discovered in the es folder, a set of huge files: java\_pid\_xxx.hprof.. Generated each time my es got 'blocked'. Now, this files are generated when the process runs out of memory. I'm running with 100G of heap memory allocated, and I expect that for large memory consuming operations the memory to be swapped. Anyhow, it seems that on my machine, each time I'm running the above query, I'm killing it. Other informations: - the jhat heap histogram can be found here: [https://gist.github.com/1497379](https://gist.github.com/1497379) - through JConsole, each time I'm executing this query, I can see how the heap memory increases !very fast! it seems that 100G are consumed in a few seconds. - regarding the node stats and cluster stats es provides: I cannot access them after es dies, but here is the info of es, right after restart when everything is okey: os: { refresh\_interval: 1000 cpu: { vendor: AMD model: Opteron mhz: 1900 total\_cores: 48 total\_sockets: 4 cores\_per\_socket: 12 cache\_size: 512b cache\_size\_in\_bytes: 512 } mem: { total: 126gb total\_in\_bytes: 135321870336 } swap: { total: 64gb total\_in\_bytes: 68803354624 } } process: { refresh\_interval: 1000 id: 15926 max\_file\_descriptors: 128000 } jvm: { pid: 15926 version: 1.6..0\_27 vm\_name: Java HotSpot(TM) 64-Bit Server VM vm\_version: 20.2-b06 vm\_vendor: Sun Microsystems Inc. start\_time: 1324275713418 mem: { heap\_init: 100gb heap\_init\_in\_bytes: 107374182400 heap\_max: 99.9gb heap\_max\_in\_bytes: 107302223872 non\_heap\_init: 23.1mb non\_heap\_init\_in\_bytes: 24313856 non\_heap\_max: 130mb non\_heap\_max\_in\_bytes: 136314880 } } Any suggestions? Tnx, Alex On Tue, Dec 13, 2011 at 11:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: ``` Also, I would add that you make sure to enable mlockall in the ``` configuration to make sure the OS will not swap the elasticsearch process. I never ran ES with 100gb of memory, whats your typical memory usage? (node stats can give you a lot of information, also on the jvm level). ``` On Tue, Dec 13, 2011 at 12:17 PM, Paul Loy ``` [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: ``` That's some machine! Yeah, with stutters like this it's very useful to know ``` what's going on with your heap (and other resources). You can watch the heap via JConsole or use some monitoring tool like traverse to fire emails when the heap gets too big or simply be able to view historical graphs of all your JMX exposed variables. ``` To enable JMX it looks like you need jmx.create_connector: ``` true in your elasticsearch.yml. ``` VisualVM also has an awesome Visual GC plugin that lets ``` you see which of the various sections of your Heap are filling up. ``` Cheers, Paul. On Tue, Dec 13, 2011 at 12:54 AM, Sisu Alexandru ``` [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) wrote: ``` Yes, the machine has 128 GB , 48 cores. And till now ``` we didn't have no problems. Anyhow, yesterday I restarted the machine, and everything got back to normal. Still I'm interesting in how to configure the jmx for es. Any tips? 🙂 On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: ``` You configured ES with 100g? Do you have a machine ``` with a 100gb of memory? How much memory does your machine has? I am surprise that it even started.... (swap might be really big). ``` On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru ``` [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) wrote: ``` Well I started up elastic search with Xms and ``` Xmx set to 100G. That should've been enough. Monitoring though JMX sounds a good ideea. but how can you configure jmx options in es? The only documentation that I found was here [http://www.elasticsearch.org/guide/reference/modules/jmx.html](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its not very clear to me what should I do. Tnx, Alex ``` On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy ``` [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: ``` Garbage Collection? How much memory are ``` you giving each JVM? If it's a large amount and you haven't tuned your GC options on the JVM, this is a likely cause. ``` I don't suppose you had something ``` monitoring JMX over that time period. If you did you'd be able to see if this was the issue if you notice the Heap Space Used dropping off. ``` Paul. On Mon, Dec 12, 2011 at 8:00 AM, Sisu ``` Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) wrote: ``` Hello all, Our es suddenly got blocked for 15 ``` minutes.. That means: it suddenly stopped handling search requests and also status requests:curl -XGET [http://localhost:9200/\_status](http://localhost:9200/_status). ES version: 0.18.5 The size of the index is arround 200G. One ES client. 20 shards. All on one single machine. No mirrors. OS: centos 5. Here is the gist: [https://gist.github.com/1467971](https://gist.github.com/1467971) This info was obtained by jstack. (jstack -F 27029 (pid of es)). It looks like all the threads got blocked (all 132 )?! The size of the jstack output is much more bigger I've copy pasted only some parts of it. Any ideas? Tnx in advance, Alex ``` -- ``` * * * ``` Paul Loy paul@keteracel.com http://uk.linkedin.com/in/paulloy -- --------------------------------------------- Paul Loy paul@keteracel.com http://uk.linkedin.com/in/paulloy ``` ---
**Author:** ![Sisu\_Alexandru](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@Sisu\_Alexandru](https://discuss.elastic.co/u/Sisu_Alexandru)\ **Post date:** [December 19, 2011, 4:07pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/11 "2011-12-19T16:07:09Z")
Hi Aurelien Tnx for the prompt answer. I run es with in foreground so that I can see the eventual OOM. And indeed it turned out to be an OOM: java.lang.OutOfMemoryError: Java heap space Dumping heap to java\_pid29451.hprof ... Exception in thread "elasticsearch[search]-pool-3-thread-18" java.lang.OutOfMemoryError: Java heap space at org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61) at org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:84) at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:52) at org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34) at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:110) at org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:119) at org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:127) at org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:71) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576) at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:199) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:383) Command line params: home/jdk1.6.0\_27/bin/java -Xms256m -Xmx1g -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=/home/elasticsearch-0.18.5 -Des-foreground=yes -cp :/home/elasticsearch-0.18.5/lib/_:/home/elasticsearch-0.18.5/lib/sigar/_ -Xms100G -Xmx100G org.elasticsearch.bootstrap.Elasticsearch The dumps size of hprof file: - currently is 70GB and is increasing (slower). Probably it will reach 100 gb. What I did right after this was to run on my development machine the same query on a smaller dataset. It seems that ResidentFieldDataCache that extends the (AbstractConcurrentMapFieldDataCache) doesnt get cleared? (I tried to put a breakpoint on clear method , and it never gets called). On the other hand, I also didn't configured my index with no caching options. I'm working now on getting the thread dumps and the verbose output of gc. On Mon, Dec 19, 2011 at 3:37 PM, Aurélien [aurelien.dehay@gmail.com](mailto:aurelien.dehay@gmail.com) wrote: > Hello. > > hprof files are generated by -XX:+HeapDumpOnOutOfMemoryError > parameter. The files should be roughly same size as heap size when OOM > occurs, but could you give us the size of those files? > > jhat profiles does not show a 100GB used heap, but do you have any OOM > error in log files? I don't know if you modified launch scripts, but > theses errors should be redirected via stderr. Do you have a huge CPU > consommation on 1 or more CPU during the "es blocked" situation? > > It should be interesting to: > > - redirect stderr and stdout to a file if it's not done already (with > &\> /path/to/file.log at the end of java command file) > - activate verbose gc to a specific file > - launch visualvm with visualgc plugin > - launch recurrent thread dumps (with kill -3 ) within short time > period (one thread dump every 10 or 15 seconds) > > and then reproduce the problem asap to avoid generating to much logs. > > If you have an OOM, you will wich generation is full with the > verbosegc and visualgc, if you have a problem with threads you will > see them in thread dumps (I use samurai [http://yusuke.homeip.net/samurai/](http://yusuke.homeip.net/samurai/) > to analyze thread dumps), MAT to analyze heap dump/hprof files (but > huge hprof should be difficult to read, maybe with jhat). Samurai can > read verbosegc files too. > > In any case, try to use a more recent JVM, please send the complete > command line of your ES, and maybe try another JDK (like jrockit with > either generation/non generational GC). > > 100GB is a quite huge memory for actual GC, see > [Understanding Java Garbage Collection and What You Can Do about It](http://www.infoq.com/presentations/Understanding-Java-Garbage-Collection) > > Rgds. > > De: "Sisu Alexandru" [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) > À: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > Envoyé: Lundi 19 Décembre 2011 15:11:30 > Objet: Re: es got blocked > > Hello all, > Following your suggestion I've tried: > > - running with bootstrap.mlockall set to True. > > - i've enabled the JMX monitoring. > The es continues to hangs. I can reproduce the problem again and > again, by executing the following query: > { "facets": { "term\_count": { "global": true, > "terms": { "field": "body", "size": 100 } } }, > "size": 0} > (I'm trying to retrieve the most common 100 terms for this field). > To my surprise, I've discovered in the es folder, a set of huge > files: java\_pid\_xxx.hprof.. Generated each time my es got 'blocked'. > Now, this files are generated when the process runs out of > memory. > I'm running with 100G of heap memory allocated, and I expect that > for large memory consuming operations the memory to be swapped. > Anyhow, it seems that on my machine, each time I'm running the > above query, I'm killing it. > Other informations: > > - the jhat heap histogram can be found here: > [jhat\_es\_hprof\_stat · GitHub](https://gist.github.com/1497379) > > - through JConsole, each time I'm executing this query, I can see > how the heap memory increases !very fast! it seems that 100G are > consumed in a few seconds. > > - regarding the node stats and cluster stats es provides: I cannot > access them after es dies, but here is the info of es, right after > restart when everything is okey: > os: { > refresh\_interval: 1000 > cpu: { > vendor: AMD > model: Opteron > mhz: 1900 > total\_cores: 48 > total\_sockets: 4 > cores\_per\_socket: 12 > cache\_size: 512b > cache\_size\_in\_bytes: 512 > } > mem: { > total: 126gb > total\_in\_bytes: 135321870336 > } > swap: { > total: 64gb > total\_in\_bytes: 68803354624 > } > } > process: { > refresh\_interval: 1000 > id: 15926 > max\_file\_descriptors: 128000 > } > jvm: { > pid: 15926 > version: 1.6..0\_27 > vm\_name: Java HotSpot(TM) 64-Bit Server VM > vm\_version: 20.2-b06 > vm\_vendor: Sun Microsystems Inc. > start\_time: 1324275713418 > mem: { > heap\_init: 100gb > heap\_init\_in\_bytes: 107374182400 > heap\_max: 99.9gb > heap\_max\_in\_bytes: 107302223872 > non\_heap\_init: 23.1mb > non\_heap\_init\_in\_bytes: 24313856 > non\_heap\_max: 130mb > non\_heap\_max\_in\_bytes: 136314880 > } > } > Any suggestions? > Tnx, > Alex > On Tue, Dec 13, 2011 at 11:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > wrote: > > what's going on with your heap (and other resources). You can watch > the heap via JConsole or use some monitoring tool like traverse to > fire emails when the heap gets too big or simply be able to view > historical graphs of all your JMX exposed variables. > > ``` > To enable JMX it looks like you need jmx.create_connector: > > ``` > > true in your elasticsearch.yml. > > ``` > VisualVM also has an awesome Visual GC plugin that lets > > ``` > > you see which of the various sections of your Heap are filling up. > > ``` > Cheers, > > Paul. > > On Tue, Dec 13, 2011 at 12:54 AM, Sisu Alexandru > > ``` > > [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) wrote: > > ``` > Yes, the machine has 128 GB , 48 cores. And till now > > ``` > > we didn't have no problems. > Anyhow, yesterday I restarted the machine, and > everything got back to normal. > Still I'm interesting in how to configure the jmx for > es. Any tips? 🙂 > On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon > [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > ``` > You configured ES with 100g? Do you have a machine > > ``` > > with a 100gb of memory? How much memory does your machine has? I am > surprise that it even started.... (swap might be really big). > > ``` > On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru > > ``` > > [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) wrote: > > ``` > Well I started up elastic search with Xms and > > ``` > > Xmx set to 100G. > That should've been enough. > Monitoring though JMX sounds a good ideea. but > how can you configure jmx options in es? > The only documentation that I found was here > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but its > not very clear to me what should I do. > Tnx, > Alex > > ``` > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy > > ``` > > [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > ``` > Garbage Collection? How much memory are > > ``` > > you giving each JVM? If it's a large amount and you haven't tuned your > GC options on the JVM, this is a likely cause. > > ``` > I don't suppose you had something > > ``` > > monitoring JMX over that time period. If you did you'd be able to see > if this was the issue if you notice the Heap Space Used dropping off. > > ``` > Paul. > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu > > ``` > > Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) wrote: > > ``` > Hello all, > Our es suddenly got blocked for 15 > > ``` > > minutes.. That means: it suddenly stopped handling search requests > and also status requests:curl -XGET [http://localhost:9200/\_status](http://localhost:9200/_status). > ES version: 0.18.5 > The size of the index is arround 200G. > One ES client. > 20 shards. All on one single machine. > No mirrors. > OS: centos 5. > Here is the gist: > [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > This info was obtained by jstack. > (jstack -F 27029 (pid of es)). It looks like all the threads got > blocked (all 132 )?! > The size of the jstack output is much > more bigger I've copy pasted only some parts of it. > Any ideas? > Tnx in advance, > Alex > > ``` > -- > > ``` > > * * * > > ``` > Paul Loy > paul@keteracel.com > http://uk.linkedin.com/in/paulloy > > -- > --------------------------------------------- > Paul Loy > paul@keteracel.com > http://uk.linkedin.com/in/paulloy > > ``` ---
**Author:** ![Aurelien\_2](https://avatars.discourse-cdn.com/v4/letter/a/a587f6/32.png) [@Aurelien\_2](https://discuss.elastic.co/u/Aurelien_2)\ **Post date:** [December 19, 2011, 4:30pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/12 "2011-12-19T16:30:18Z")
Hi. don't bother taking the thread dumps, it's a clear OOM, TD won't be really helpful. I've never use jhat to do OOM analysis, but I don"t know if MAT [Eclipse Memory Analyzer Open Source Project | The Eclipse Foundation](http://eclipse.org/mat/) will handle a hprof of 100GB. Worth a try on a machine with a lot of memory and a well customized eclipse. Furthermore, if ES works on Java 7, I would try to use it, and simplify the command line by removing firsts Xms Xmx and all the XX parameters. It will certainly not solve the OOM, but the new G1 GC may be more efficient on large heap size. I will let other people answer on code specific, I'm not a java dev. Regards. On 19 déc, 17:07, Sisu Alexandru [sisu.eu...@gmail.com](mailto:sisu.eu...@gmail.com) wrote: > Hi Aurelien > > Tnx for the prompt answer. > I run es with in foreground so that I can see the eventual OOM. And indeed > it turned out to be an OOM: > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java\_pid29451.hprof ... > Exception in thread "elasticsearch[search]-pool-3-thread-18" > java.lang.OutOfMemoryError: Java heap space > at > org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:61) > at > org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:84) > at > org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:52) > at > org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34) > at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:110) > at > org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:119) > at > org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:127) > at > org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:71) > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576) > at > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:199) > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:383) > > Command line params: > > home/jdk1.6.0\_27/bin/java -Xms256m -Xmx1g -Xss128k -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError > -Delasticsearch -Des.path.home=/home/elasticsearch-0.18.5 > -Des-foreground=yes -cp > :/home/elasticsearch-0.18.5/lib/_:/home/elasticsearch-0.18.5/lib/sigar/_ > -Xms100G -Xmx100G org.elasticsearch.bootstrap.Elasticsearch > > The dumps size of hprof file: > > - currently is 70GB and is increasing (slower). Probably it will reach 100 > gb. > > What I did right after this was to run on my development machine the same > query on a smaller dataset. > It seems that ResidentFieldDataCache that extends the > (AbstractConcurrentMapFieldDataCache) doesnt get cleared? (I tried to put a > breakpoint on clear method , and it never gets called). > On the other hand, I also didn't configured my index with no caching > options. > > I'm working now on getting the thread dumps and the verbose output of gc. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 19, 2011, 6:40pm UTC](https://discuss.elastic.co/t/es-got-blocked/6129/13 "2011-12-19T18:40:52Z")
It seems like you are trying to get terms on a field (body) that has many of those (guessing by the name of it), resulting in the excessive memory usage and OOM. The terms facet is not designed to be used on fields with many terms. On Mon, Dec 19, 2011 at 4:11 PM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > Hello all, > > Following your suggestion I've tried: > > - running with bootstrap.mlockall set to True. > - i've enabled the JMX monitoring. > > The es continues to hangs. I can reproduce the problem again and again, by > executing the following query: > { "facets": { "term\_count": { "global": true, "terms": { > "field": "body", "size": 100 } } }, "size": 0} > > (I'm trying to retrieve the most common 100 terms for this field). > > To my surprise, I've discovered in the es folder, a set of huge files:\* > java\_pid\__xxx_.hprof\*. Generated each time my es got\* 'blocked'.\* > Now, this files are generated when the process runs out of memory. > > I'm running with 100G of heap memory allocated, and I expect that for > large memory consuming operations the memory to be swapped. > > Anyhow, it seems that on my machine, each time I'm running the above > query, I'm killing it. > > Other informations: > > - the jhat heap histogram can be found here: > [jhat\_es\_hprof\_stat · GitHub](https://gist.github.com/1497379) > - through _JConsole_, each time I'm executing this query, I can see how > the heap memory increases !very fast! it seems that 100G are consumed in a > few seconds. > - regarding the node stats and cluster stats es provides: I cannot access > them after es dies, but here is the info of es, right after restart when > everything is okey: > os: { > refresh\_interval: 1000 > cpu: { > vendor: AMD > model: Opteron > mhz: 1900 > total\_cores: 48 > total\_sockets: 4 > cores\_per\_socket: 12 > cache\_size: 512b > cache\_size\_in\_bytes: 512 > } > mem: { > total: 126gb > total\_in\_bytes: 135321870336 > } > swap: { > total: 64gb > total\_in\_bytes: 68803354624 > } > } > process: { > refresh\_interval: 1000 > id: 15926 > max\_file\_descriptors: 128000 > } > jvm: { > pid: 15926 > version: 1.6.0\_27 > vm\_name: Java HotSpot(TM) 64-Bit Server VM > vm\_version: 20.2-b06 > vm\_vendor: Sun Microsystems Inc. > start\_time: 1324275713418 > mem: { > heap\_init: 100gb > heap\_init\_in\_bytes: 107374182400 > heap\_max: 99.9gb > heap\_max\_in\_bytes: 107302223872 > non\_heap\_init: 23.1mb > non\_heap\_init\_in\_bytes: 24313856 > non\_heap\_max: 130mb > non\_heap\_max\_in\_bytes: 136314880 > } > } > > Any suggestions? > > Tnx, > Alex > > On Tue, Dec 13, 2011 at 11:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Also, I would add that you make sure to enable mlockall in the > > configuration to make sure the OS will not swap the elasticsearch process. > > I never ran ES with 100gb of memory, whats your typical memory usage? (node > > stats can give you a lot of information, also on the jvm level). > > > > On Tue, Dec 13, 2011 at 12:17 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > That's some machine! > > > > > > Yeah, with stutters like this it's very useful to know what's going on > > > with your heap (and other resources). You can watch the heap via JConsole > > > or use some monitoring tool like traverse to fire emails when the heap gets > > > too big or simply be able to view historical graphs of all your JMX exposed > > > variables. > > > > > > To enable JMX it looks like you need jmx.create\_connector: true in your > > > elasticsearch.yml. > > > > > > VisualVM also has an awesome Visual GC plugin that lets you see which of > > > the various sections of your Heap are filling up. > > > > > > Cheers, > > > > > > Paul. > > > > > > On Tue, Dec 13, 2011 at 12:54 AM, Sisu Alexandru [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)wrote: > > > > > > > Yes, the machine has 128 GB , 48 cores. And till now we didn't have no > > > > problems. > > > > Anyhow, yesterday I restarted the machine, and everything got back to > > > > normal. > > > > Still I'm interesting in how to configure the jmx for es. Any tips? 🙂 > > > > > > > > On Mon, Dec 12, 2011 at 11:43 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > You configured ES with 100g? Do you have a machine with a 100gb of > > > > > memory? How much memory does your machine has? I am surprise that it even > > > > > started... (swap might be really big). > > > > > > > > > > On Mon, Dec 12, 2011 at 11:12 PM, Sisu Alexandru \<[sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com) > > > > > > > > > > > wrote: > > > > > > > > > > > Well I started up Elasticsearch with Xms and Xmx set to 100G. > > > > > > That should've been enough. > > > > > > > > > > > > Monitoring though JMX sounds a good ideea. but how can you configure > > > > > > jmx options in es? > > > > > > The only documentation that I found was here > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/jmx.html) but > > > > > > its not very clear to me what should I do. > > > > > > > > > > > > Tnx, > > > > > > > > > > > > Alex > > > > > > > > > > > > On Mon, Dec 12, 2011 at 5:40 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com)wrote: > > > > > > > > > > > > > Garbage Collection? How much memory are you giving each JVM? If it's > > > > > > > a large amount and you haven't tuned your GC options on the JVM, this is a > > > > > > > likely cause. > > > > > > > > > > > > > > I don't suppose you had something monitoring JMX over that time > > > > > > > period. If you did you'd be able to see if this was the issue if you notice > > > > > > > the Heap Space Used dropping off. > > > > > > > > > > > > > > Paul. > > > > > > > > > > > > > > On Mon, Dec 12, 2011 at 8:00 AM, Sisu Alexandru \< > > > > > > > [sisu.eugen@gmail.com](mailto:sisu.eugen@gmail.com)\> wrote: > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > > > Our es suddenly got blocked for 15 minutes. That means: it > > > > > > > > suddenly stopped handling search requests and also status requests:curl > > > > > > > > -XGET [http://localhost:9200/\_status](http://localhost:9200/_status). > > > > > > > > > > > > > > > > ES version: 0.18.5 > > > > > > > > The size of the index is arround 200G. > > > > > > > > One ES client. > > > > > > > > 20 shards. All on one single machine. > > > > > > > > No mirrors. > > > > > > > > OS: centos 5. > > > > > > > > > > > > > > > > Here is the gist: [jstack elastic\_search · GitHub](https://gist.github.com/1467971) > > > > > > > > This info was obtained by jstack. (jstack -F 27029 (pid of es)). It > > > > > > > > looks like all the threads got blocked (all 132 )?! > > > > > > > > > > > > > > > > The size of the jstack output is much more bigger I've copy pasted > > > > > > > > only some parts of it. > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > Tnx in advance, > > > > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > ## -- > > > > > > > > > > > > > > Paul Loy > > > > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > > > > > ## -- > > > > > > Paul Loy > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:44am UTC](https://discuss.elastic.co/t/es-got-blocked/6129/14 "2017-07-06T03:44:57Z")