# 100% processor usage

**URL:** <https://discuss.elastic.co/t/100-processor-usage/8749>\
**Category:** Elasticsearch\
**Created:** [August 15, 2012, 3:10pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749 "2012-08-15T15:10:35Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 15, 2012, 3:10pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/1 "2012-08-15T15:10:35Z")

</div>

Hi,

I have the following problem. I have 2 nodes. I started importing data into  
one of them (node A). After few hours of importing process the node B  
started using 100% processor and hardly answered other search and insert  
requests. It imported 11GB of data. How can I investigate such problems?  
Are there any logs that can help with such situation? What should I do ?

Thank you for help.

Best regards.  
Marcin Dojwa.

--

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 15, 2012, 3:18pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/2 "2012-08-15T15:18:07Z")

</div>

Have you looked at the Leap second problem?

[http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)

## David

Le 15 août 2012 à 17:10, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) a écrit :

> Hi,
> 
> I have the following problem. I have 2 nodes. I started importing data into one of them (node A). After few hours of importing process the node B started using 100% processor and hardly answered other search and insert requests. It imported 11GB of data. How can I investigate such problems? Are there any logs that can help with such situation? What should I do ?
> 
> Thank you for help.
> 
> ## Best regards. Marcin Dojwa.

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 15, 2012, 3:33pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/3 "2012-08-15T15:33:06Z")

</div>

No, I will check this, thanks.

2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)

> Have you looked at the Leap second problem?
> 
> [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> 
> ## David
> 
> Le 15 août 2012 à 17:10, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) a écrit :
> 
> Hi,
> 
> I have the following problem. I have 2 nodes. I started importing data  
> into one of them (node A). After few hours of importing process the node B  
> started using 100% processor and hardly answered other search and insert  
> requests. It imported 11GB of data. How can I investigate such problems?  
> Are there any logs that can help with such situation? What should I do ?
> 
> Thank you for help.
> 
> Best regards.  
> Marcin Dojwa.
> 
> --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 15, 2012, 3:33pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/4 "2012-08-15T15:33:55Z")

</div>

But my question is still open, how to investigate such problems ? 🙂

Best regards.

2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> No, I will check this, thanks.
> 
> 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)
> 
> > Have you looked at the Leap second problem?
> > 
> > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > 
> > ## David
> > 
> > Le 15 août 2012 à 17:10, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) a écrit :
> > 
> > Hi,
> > 
> > I have the following problem. I have 2 nodes. I started importing data  
> > into one of them (node A). After few hours of importing process the node B  
> > started using 100% processor and hardly answered other search and insert  
> > requests. It imported 11GB of data. How can I investigate such problems?  
> > Are there any logs that can help with such situation? What should I do ?
> > 
> > Thank you for help.
> > 
> > Best regards.  
> > Marcin Dojwa.
> > 
> > --
> > 
> > --

--

---

<div class="post-metadata">

**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\
**Post date:** [August 15, 2012, 9:32pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/5 "2012-08-15T21:32:47Z")

</div>

Hello Marcin!

Did you query ElasticSearch or was it only indexing data ?

Anyway, first of all I would enable GC logging in ElasticSearch, to be  
sure that the problems you are experiencing are not garbage collection  
related.

There are some tools that can help you diagnosing what is happening  
with your nodes - look at the front ends mentioned on ElasticSearch  
clients page ([http://www.elasticsearch.org/guide/appendix/clients.html](http://www.elasticsearch.org/guide/appendix/clients.html)).  
Of course you can use the ones that are available on your operating  
system like vmstat or dstat.

--  
Regards,  
Rafał Kuć  
Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - ElasticSearch

But my question is still open, how to investigate such problems ? 🙂

Best regards.

2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
No, I will check this, thanks.

2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
Have you looked at the Leap second problem?

[http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)

## David

Le 15 août 2012 à 17:10, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) a écrit :

Hi,

I have the following problem. I have 2 nodes. I started importing data into one of them (node A). After few hours of importing process the node B started using 100% processor and hardly answered other search and insert requests. It imported 11GB of data. How can I investigate such problems? Are there any logs that can help with such situation? What should I do ?

Thank you for help.

## Best regards. Marcin Dojwa.

--

--

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 16, 2012, 5:22am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/6 "2012-08-16T05:22:27Z")

</div>

Hi,

Thank you very much for your help, I will check this. Could you tell me how  
to enable GC logging in ES ? Thanks.

Best regards.

2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)

> Hello Marcin!
> 
> Did you query Elasticsearch or was it only indexing data ?
> 
> Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> sure that the problems you are experiencing are not garbage collection  
> related.
> 
> There are some tools that can help you diagnosing what is happening  
> with your nodes - look at the front ends mentioned on Elasticsearch  
> clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> Of course you can use the ones that are available on your operating  
> system like vmstat or dstat.
> 
> --  
> Regards,  
> Rafał Kuć  
> Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch
> 
> But my question is still open, how to investigate such problems ? 🙂
> 
> Best regards.
> 
> 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> No, I will check this, thanks.
> 
> 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> Have you looked at the Leap second problem?
> 
> [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> 
> ## David
> 
> Le 15 août 2012 à 17:10, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) a écrit :
> 
> Hi,
> 
> I have the following problem. I have 2 nodes. I started importing data  
> into one of them (node A). After few hours of importing process the node B  
> started using 100% processor and hardly answered other search and insert  
> requests. It imported 11GB of data. How can I investigate such problems?  
> Are there any logs that can help with such situation? What should I do ?
> 
> Thank you for help.
> 
> ## Best regards. Marcin Dojwa.
> 
> --
> 
> --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\
**Post date:** [August 16, 2012, 7:27am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/7 "2012-08-16T07:27:10Z")

</div>

Hello!

You should have the following section in your elasticsearch.yml file:

################################## GC Logging ################################

#monitor.jvm.gc.ParNew.warn: 1000ms

#monitor.jvm.gc.ParNew.info: 700ms

#monitor.jvm.gc.ParNew.debug: 400ms

#monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s

#monitor.jvm.gc.ConcurrentMarkSweep.info: 5s

#monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s

Just remove the comment character, restart ElasticSearch and you will have that being logged.

_--_

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch - ElasticSearch_

Hi,

Thank you very much for your help, I will check this. Could you tell me how to enable GC logging in ES ? Thanks.

Best regards.

2012/8/15 Rafał Kuć \<[r.kuc@solr.pl](mailto:r.kuc@solr.pl)\>

Hello Marcin!

Did you query ElasticSearch or was it only indexing data ?

Anyway, first of all I would enable GC logging in ElasticSearch, to be

sure that the problems you are experiencing are not garbage collection

related.

There are some tools that can help you diagnosing what is happening

with your nodes - look at the front ends mentioned on ElasticSearch

clients page ([http://www.elasticsearch.org/guide/appendix/clients.html](http://www.elasticsearch.org/guide/appendix/clients.html)).

Of course you can use the ones that are available on your operating

system like vmstat or dstat.

--

Regards,

Rafał Kuć

Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - ElasticSearch

But my question is still open, how to investigate such problems ? 🙂

Best regards.

2012/8/15 Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)\>

No, I will check this, thanks.

2012/8/15 David Pilato \<[david@pilato.fr](mailto:david@pilato.fr)\>

Have you looked at the Leap second problem?

[http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)

David

--

Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)\> a écrit :

Hi,

I have the following problem. I have 2 nodes. I started importing data into one of them (node A). After few hours of importing process the node B started using 100% processor and hardly answered other search and insert requests. It imported 11GB of data. How can I investigate such problems? Are there any logs that can help with such situation? What should I do ?

Thank you for help.

Best regards.

Marcin Dojwa.

--

--

--

--

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [August 16, 2012, 7:37am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/8 "2012-08-16T07:37:45Z")

</div>

Thank you Rafał.

Best regards  
Marcin Dojwa

2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)

> Hello!
> 
> You should have the following section in your elasticsearch.yml file:
> 
> ################################## GC Logging  
> ################################
> 
> #monitor.jvm.gc.ParNew.warn: 1000ms  
> #monitor.jvm.gc.ParNew.info: 700ms  
> #monitor.jvm.gc.ParNew.debug: 400ms
> 
> #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> 
> Just remove the comment character, restart Elasticsearch and you will have  
> that being logged.
> 
> _--  
> Regards,  
> Rafał Kuć  
> Sematext :: \*  
> [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> 
> - 
> 
> Hi,
> 
> Thank you very much for your help, I will check this. Could you tell me  
> how to enable GC logging in ES ? Thanks.
> 
> Best regards.
> 
> 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> Hello Marcin!
> 
> Did you query Elasticsearch or was it only indexing data ?
> 
> Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> sure that the problems you are experiencing are not garbage collection  
> related.
> 
> There are some tools that can help you diagnosing what is happening  
> with your nodes - look at the front ends mentioned on Elasticsearch  
> clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> Of course you can use the ones that are available on your operating  
> system like vmstat or dstat.
> 
> --  
> Regards,  
> Rafał Kuć  
> Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch
> 
> But my question is still open, how to investigate such problems ? 🙂
> 
> Best regards.
> 
> 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> No, I will check this, thanks.
> 
> 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> Have you looked at the Leap second problem?
> 
> [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> 
> ## David
> 
> Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > a écrit :
> 
> Hi,
> 
> I have the following problem. I have 2 nodes. I started importing data  
> into one of them (node A). After few hours of importing process the node B  
> started using 100% processor and hardly answered other search and insert  
> requests. It imported 11GB of data. How can I investigate such problems?  
> Are there any logs that can help with such situation? What should I do ?
> 
> Thank you for help.
> 
> ## Best regards. Marcin Dojwa.
> 
> --
> 
> --
> 
> --
> 
> --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [September 27, 2012, 9:23am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/9 "2012-09-27T09:23:50Z")

</div>

Hi,

I still need help in this 🙂 I have the following information:

1. 2 nodes (es1 and es2)
2. 1 index Users with 30 shards and 1 replica

I do the following:

1. On es2 I run consecutively many bulk index operations (each bulk has  
1000 inserts) and I do about 1000 such bulk inserts.
2. At some point of inserting es1 uses about 300% of processor and bulk  
insert hangs. Then I stop it.
3. After 1 day doing nothing with any of ES nodes (I turn of everything  
that uses ES) es1 uses 100% of processor and es2 is OK.  
es1:  
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java

es2:  
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java  
4. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
{

- cluster\_name: "production",
- status: "green",
- timed\_out: false,
- number\_of\_nodes: 2,
- number\_of\_data\_nodes: 2,
- active\_primary\_shards: 30,
- active\_shards: 60,
- relocating\_shards: 0,
- initializing\_shards: 0,
- unassigned\_shards: 0

}  
5. es1: vmstat 5 5  
procs -----------memory---------- ---swap-- -----io---- -system--  
----cpu----  
r b swpd free buff cache si so bi bo in cs us sy id  
wa  
1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1 0  
99 0  
1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16 0  
84 0  
1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17 0  
83 0  
1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16 0  
84 0  
1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18 0  
82 0

1. es2: vmstat 5 5  
procs -----------memory---------- ---swap-- -----io---- -system--  
----cpu----  
r b swpd free buff cache si so bi bo in cs us sy id  
wa  
0 0 0 11327116 253432 1299284 0 0 1 13 16 16 0 0  
100 0  
0 0 0 11326984 253432 1299284 0 0 0 0 897 2154 0 0  
100 0  
0 0 0 11326984 253432 1299284 0 0 0 7 914 2174 0 0  
100 0  
0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225 0 0  
100 0  
0 0 0 11326984 253432 1299284 0 0 0 7 874 2109 0 0  
100 0

2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows that  
everything is OK

3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
everything is OK and it shows that:  
OS - CPU:  
Total: 100%  
User: 15%  
Sys: 0%

OS - LOAD:  
2: 1.11  
1: 1.24  
0: 1.24

Process - CPU:  
Total: 400%  
Process: 123%

1. After restarting elasticsearch on es1 and stabilizing es1 works fine but  
the same situation happens on es2 (100% processor usage). The only way to  
fix that is stopping all ES nodes and starting them again.

Is there anything I can do to investigate this problem?

Thanks for help.

Best regards.

2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> Thank you Rafał.
> 
> Best regards  
> Marcin Dojwa
> 
> 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> 
> > Hello!
> > 
> > You should have the following section in your elasticsearch.yml file:
> > 
> > ################################## GC Logging  
> > ################################
> > 
> > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > #monitor.jvm.gc.ParNew.info: 700ms  
> > #monitor.jvm.gc.ParNew.debug: 400ms
> > 
> > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > 
> > Just remove the comment character, restart Elasticsearch and you will  
> > have that being logged.
> > 
> > _--  
> > Regards,  
> > Rafał Kuć  
> > Sematext :: \*  
> > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > 
> > - 
> > 
> > Hi,
> > 
> > Thank you very much for your help, I will check this. Could you tell me  
> > how to enable GC logging in ES ? Thanks.
> > 
> > Best regards.
> > 
> > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > Hello Marcin!
> > 
> > Did you query Elasticsearch or was it only indexing data ?
> > 
> > Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> > sure that the problems you are experiencing are not garbage collection  
> > related.
> > 
> > There are some tools that can help you diagnosing what is happening  
> > with your nodes - look at the front ends mentioned on Elasticsearch  
> > clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > Of course you can use the ones that are available on your operating  
> > system like vmstat or dstat.
> > 
> > --  
> > Regards,  
> > Rafał Kuć  
> > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > Elasticsearch
> > 
> > But my question is still open, how to investigate such problems ? 🙂
> > 
> > Best regards.
> > 
> > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > No, I will check this, thanks.
> > 
> > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > Have you looked at the Leap second problem?
> > 
> > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > 
> > ## David
> > 
> > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > a écrit :
> > 
> > Hi,
> > 
> > I have the following problem. I have 2 nodes. I started importing data  
> > into one of them (node A). After few hours of importing process the node B  
> > started using 100% processor and hardly answered other search and insert  
> > requests. It imported 11GB of data. How can I investigate such problems?  
> > Are there any logs that can help with such situation? What should I do ?
> > 
> > Thank you for help.
> > 
> > ## Best regards. Marcin Dojwa.
> > 
> > --
> > 
> > --
> > 
> > --
> > 
> > --
> > 
> > --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [September 27, 2012, 9:42am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/10 "2012-09-27T09:42:13Z")

</div>

I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure if  
that will help.

2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> Hi,
> 
> I still need help in this 🙂 I have the following information:
> 
> 1. 2 nodes (es1 and es2)
> 2. 1 index Users with 30 shards and 1 replica
> 
> I do the following:
> 
> 1. On es2 I run consecutively many bulk index operations (each bulk has  
> 1000 inserts) and I do about 1000 such bulk inserts.
> 2. At some point of inserting es1 uses about 300% of processor and bulk  
> insert hangs. Then I stop it.
> 3. After 1 day doing nothing with any of ES nodes (I turn of everything  
> that uses ES) es1 uses 100% of processor and es2 is OK.  
> es1:  
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 
> 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> 
> es2:  
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 
> 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java  
> 4. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> {
> 
> - cluster\_name: "production",
> - status: "green",
> - timed\_out: false,
> - number\_of\_nodes: 2,
> - number\_of\_data\_nodes: 2,
> - active\_primary\_shards: 30,
> - active\_shards: 60,
> - relocating\_shards: 0,
> - initializing\_shards: 0,
> - unassigned\_shards: 0
> 
> }  
> 5. es1: vmstat 5 5  
> procs -----------memory---------- ---swap-- -----io---- -system--  
> ----cpu----  
> r b swpd free buff cache si so bi bo in cs us sy id  
> wa  
> 1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1 0  
> 99 0  
> 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16 0  
> 84 0  
> 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17 0  
> 83 0  
> 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16 0  
> 84 0  
> 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18 0  
> 82 0
> 
> 1. es2: vmstat 5 5  
> procs -----------memory---------- ---swap-- -----io---- -system--  
> ----cpu----  
> r b swpd free buff cache si so bi bo in cs us sy id  
> wa  
> 0 0 0 11327116 253432 1299284 0 0 1 13 16 16 0 0  
> 100 0  
> 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154 0 0  
> 100 0  
> 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174 0 0  
> 100 0  
> 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225 0 0  
> 100 0  
> 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109 0 0  
> 100 0
> 
> 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows that  
> everything is OK
> 
> 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> everything is OK and it shows that:  
> OS - CPU:  
> Total: 100%  
> User: 15%  
> Sys: 0%
> 
> OS - LOAD:  
> 2: 1.11  
> 1: 1.24  
> 0: 1.24
> 
> Process - CPU:  
> Total: 400%  
> Process: 123%
> 
> 1. After restarting elasticsearch on es1 and stabilizing es1 works fine  
> but the same situation happens on es2 (100% processor usage). The only way  
> to fix that is stopping all ES nodes and starting them again.
> 
> Is there anything I can do to investigate this problem?
> 
> Thanks for help.
> 
> Best regards.
> 
> 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > Thank you Rafał.
> > 
> > Best regards  
> > Marcin Dojwa
> > 
> > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > 
> > > Hello!
> > > 
> > > You should have the following section in your elasticsearch.yml file:
> > > 
> > > ################################## GC Logging  
> > > ################################
> > > 
> > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > #monitor.jvm.gc.ParNew.info: 700ms  
> > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > 
> > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > 
> > > Just remove the comment character, restart Elasticsearch and you will  
> > > have that being logged.
> > > 
> > > _--  
> > > Regards,  
> > > Rafał Kuć  
> > > Sematext :: \*  
> > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > 
> > > - 
> > > 
> > > Hi,
> > > 
> > > Thank you very much for your help, I will check this. Could you tell me  
> > > how to enable GC logging in ES ? Thanks.
> > > 
> > > Best regards.
> > > 
> > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > Hello Marcin!
> > > 
> > > Did you query Elasticsearch or was it only indexing data ?
> > > 
> > > Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> > > sure that the problems you are experiencing are not garbage collection  
> > > related.
> > > 
> > > There are some tools that can help you diagnosing what is happening  
> > > with your nodes - look at the front ends mentioned on Elasticsearch  
> > > clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > > Of course you can use the ones that are available on your operating  
> > > system like vmstat or dstat.
> > > 
> > > --  
> > > Regards,  
> > > Rafał Kuć  
> > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > Elasticsearch
> > > 
> > > But my question is still open, how to investigate such problems ? 🙂
> > > 
> > > Best regards.
> > > 
> > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > No, I will check this, thanks.
> > > 
> > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > Have you looked at the Leap second problem?
> > > 
> > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > 
> > > ## David
> > > 
> > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > 
> > > > a écrit :
> > > 
> > > Hi,
> > > 
> > > I have the following problem. I have 2 nodes. I started importing data  
> > > into one of them (node A). After few hours of importing process the node B  
> > > started using 100% processor and hardly answered other search and insert  
> > > requests. It imported 11GB of data. How can I investigate such problems?  
> > > Are there any logs that can help with such situation? What should I do ?
> > > 
> > > Thank you for help.
> > > 
> > > ## Best regards. Marcin Dojwa.
> > > 
> > > --
> > > 
> > > --
> > > 
> > > --
> > > 
> > > --
> > > 
> > > --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [September 27, 2012, 10:13am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/11 "2012-09-27T10:13:49Z")

</div>

Oh, and there is nothing in gc log.

2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure if  
> that will help.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > Hi,
> > 
> > I still need help in this 🙂 I have the following information:
> > 
> > 1. 2 nodes (es1 and es2)
> > 2. 1 index Users with 30 shards and 1 replica
> > 
> > I do the following:
> > 
> > 1. On es2 I run consecutively many bulk index operations (each bulk has  
> > 1000 inserts) and I do about 1000 such bulk inserts.
> > 2. At some point of inserting es1 uses about 300% of processor and bulk  
> > insert hangs. Then I stop it.
> > 3. After 1 day doing nothing with any of ES nodes (I turn of everything  
> > that uses ES) es1 uses 100% of processor and es2 is OK.  
> > es1:  
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 
> > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> > 
> > es2:  
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 
> > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java  
> > 4. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > {
> > 
> > - cluster\_name: "production",
> > - status: "green",
> > - timed\_out: false,
> > - number\_of\_nodes: 2,
> > - number\_of\_data\_nodes: 2,
> > - active\_primary\_shards: 30,
> > - active\_shards: 60,
> > - relocating\_shards: 0,
> > - initializing\_shards: 0,
> > - unassigned\_shards: 0
> > 
> > }  
> > 5. es1: vmstat 5 5  
> > procs -----------memory---------- ---swap-- -----io---- -system--  
> > ----cpu----  
> > r b swpd free buff cache si so bi bo in cs us sy  
> > id wa  
> > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1 0  
> > 99 0  
> > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16 0  
> > 84 0  
> > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17 0  
> > 83 0  
> > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16 0  
> > 84 0  
> > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18 0  
> > 82 0
> > 
> > 1. es2: vmstat 5 5  
> > procs -----------memory---------- ---swap-- -----io---- -system--  
> > ----cpu----  
> > r b swpd free buff cache si so bi bo in cs us sy  
> > id wa  
> > 0 0 0 11327116 253432 1299284 0 0 1 13 16 16 0  
> > 0 100 0  
> > 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154 0  
> > 0 100 0  
> > 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174 0  
> > 0 100 0  
> > 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225 0  
> > 0 100 0  
> > 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109 0  
> > 0 100 0
> > 
> > 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows that  
> > everything is OK
> > 
> > 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> > everything is OK and it shows that:  
> > OS - CPU:  
> > Total: 100%  
> > User: 15%  
> > Sys: 0%
> > 
> > OS - LOAD:  
> > 2: 1.11  
> > 1: 1.24  
> > 0: 1.24
> > 
> > Process - CPU:  
> > Total: 400%  
> > Process: 123%
> > 
> > 1. After restarting elasticsearch on es1 and stabilizing es1 works fine  
> > but the same situation happens on es2 (100% processor usage). The only way  
> > to fix that is stopping all ES nodes and starting them again.
> > 
> > Is there anything I can do to investigate this problem?
> > 
> > Thanks for help.
> > 
> > Best regards.
> > 
> > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > Thank you Rafał.
> > > 
> > > Best regards  
> > > Marcin Dojwa
> > > 
> > > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > > 
> > > > Hello!
> > > > 
> > > > You should have the following section in your elasticsearch.yml file:
> > > > 
> > > > ################################## GC Logging  
> > > > ################################
> > > > 
> > > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > > #monitor.jvm.gc.ParNew.info: 700ms  
> > > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > > 
> > > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > > #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> > > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > > 
> > > > Just remove the comment character, restart Elasticsearch and you will  
> > > > have that being logged.
> > > > 
> > > > _--  
> > > > Regards,  
> > > > Rafał Kuć  
> > > > Sematext :: \*  
> > > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > > 
> > > > - 
> > > > 
> > > > Hi,
> > > > 
> > > > Thank you very much for your help, I will check this. Could you tell me  
> > > > how to enable GC logging in ES ? Thanks.
> > > > 
> > > > Best regards.
> > > > 
> > > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > > Hello Marcin!
> > > > 
> > > > Did you query Elasticsearch or was it only indexing data ?
> > > > 
> > > > Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> > > > sure that the problems you are experiencing are not garbage collection  
> > > > related.
> > > > 
> > > > There are some tools that can help you diagnosing what is happening  
> > > > with your nodes - look at the front ends mentioned on Elasticsearch  
> > > > clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)  
> > > > ).  
> > > > Of course you can use the ones that are available on your operating  
> > > > system like vmstat or dstat.
> > > > 
> > > > --  
> > > > Regards,  
> > > > Rafał Kuć  
> > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > > Elasticsearch
> > > > 
> > > > But my question is still open, how to investigate such problems ? 🙂
> > > > 
> > > > Best regards.
> > > > 
> > > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > No, I will check this, thanks.
> > > > 
> > > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > > Have you looked at the Leap second problem?
> > > > 
> > > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > > 
> > > > ## David
> > > > 
> > > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > 
> > > > > a écrit :
> > > > 
> > > > Hi,
> > > > 
> > > > I have the following problem. I have 2 nodes. I started importing data  
> > > > into one of them (node A). After few hours of importing process the node B  
> > > > started using 100% processor and hardly answered other search and insert  
> > > > requests. It imported 11GB of data. How can I investigate such problems?  
> > > > Are there any logs that can help with such situation? What should I do ?
> > > > 
> > > > Thank you for help.
> > > > 
> > > > ## Best regards. Marcin Dojwa.
> > > > 
> > > > --
> > > > 
> > > > --
> > > > 
> > > > --
> > > > 
> > > > --
> > > > 
> > > > --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [September 27, 2012, 11:00am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/12 "2012-09-27T11:00:35Z")

</div>

Fix for 1.: each bulk has 10000 inserts

Best regards.

2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> Oh, and there is nothing in gc log.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure  
> > if that will help.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > Hi,
> > > 
> > > I still need help in this 🙂 I have the following information:
> > > 
> > > 1. 2 nodes (es1 and es2)
> > > 2. 1 index Users with 30 shards and 1 replica
> > > 
> > > I do the following:
> > > 
> > > 1. On es2 I run consecutively many bulk index operations (each bulk has  
> > > 1000 inserts) and I do about 1000 such bulk inserts.
> > > 2. At some point of inserting es1 uses about 300% of processor and bulk  
> > > insert hangs. Then I stop it.
> > > 3. After 1 day doing nothing with any of ES nodes (I turn of everything  
> > > that uses ES) es1 uses 100% of processor and es2 is OK.  
> > > es1:  
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > 
> > > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> > > 
> > > es2:  
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > 
> > > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java  
> > > 4. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > > {
> > > 
> > > - cluster\_name: "production",
> > > - status: "green",
> > > - timed\_out: false,
> > > - number\_of\_nodes: 2,
> > > - number\_of\_data\_nodes: 2,
> > > - active\_primary\_shards: 30,
> > > - active\_shards: 60,
> > > - relocating\_shards: 0,
> > > - initializing\_shards: 0,
> > > - unassigned\_shards: 0
> > > 
> > > }  
> > > 5. es1: vmstat 5 5  
> > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > ----cpu----  
> > > r b swpd free buff cache si so bi bo in cs us sy  
> > > id wa  
> > > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1  
> > > 0 99 0  
> > > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16  
> > > 0 84 0  
> > > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17  
> > > 0 83 0  
> > > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16  
> > > 0 84 0  
> > > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18  
> > > 0 82 0
> > > 
> > > 1. es2: vmstat 5 5  
> > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > ----cpu----  
> > > r b swpd free buff cache si so bi bo in cs us sy  
> > > id wa  
> > > 0 0 0 11327116 253432 1299284 0 0 1 13 16 16 0  
> > > 0 100 0  
> > > 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154 0  
> > > 0 100 0  
> > > 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174 0  
> > > 0 100 0  
> > > 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225 0  
> > > 0 100 0  
> > > 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109 0  
> > > 0 100 0
> > > 
> > > 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows that  
> > > everything is OK
> > > 
> > > 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> > > everything is OK and it shows that:  
> > > OS - CPU:  
> > > Total: 100%  
> > > User: 15%  
> > > Sys: 0%
> > > 
> > > OS - LOAD:  
> > > 2: 1.11  
> > > 1: 1.24  
> > > 0: 1.24
> > > 
> > > Process - CPU:  
> > > Total: 400%  
> > > Process: 123%
> > > 
> > > 1. After restarting elasticsearch on es1 and stabilizing es1 works fine  
> > > but the same situation happens on es2 (100% processor usage). The only way  
> > > to fix that is stopping all ES nodes and starting them again.
> > > 
> > > Is there anything I can do to investigate this problem?
> > > 
> > > Thanks for help.
> > > 
> > > Best regards.
> > > 
> > > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > 
> > > > Thank you Rafał.
> > > > 
> > > > Best regards  
> > > > Marcin Dojwa
> > > > 
> > > > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > > > 
> > > > > Hello!
> > > > > 
> > > > > You should have the following section in your elasticsearch.yml file:
> > > > > 
> > > > > ################################## GC Logging  
> > > > > ################################
> > > > > 
> > > > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > > > #monitor.jvm.gc.ParNew.info: 700ms  
> > > > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > > > 
> > > > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > > > #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> > > > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > > > 
> > > > > Just remove the comment character, restart Elasticsearch and you will  
> > > > > have that being logged.
> > > > > 
> > > > > _--  
> > > > > Regards,  
> > > > > Rafał Kuć  
> > > > > Sematext :: \*  
> > > > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > > > 
> > > > > - 
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > Thank you very much for your help, I will check this. Could you tell  
> > > > > me how to enable GC logging in ES ? Thanks.
> > > > > 
> > > > > Best regards.
> > > > > 
> > > > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > > > Hello Marcin!
> > > > > 
> > > > > Did you query Elasticsearch or was it only indexing data ?
> > > > > 
> > > > > Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> > > > > sure that the problems you are experiencing are not garbage collection  
> > > > > related.
> > > > > 
> > > > > There are some tools that can help you diagnosing what is happening  
> > > > > with your nodes - look at the front ends mentioned on Elasticsearch  
> > > > > clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)  
> > > > > ).  
> > > > > Of course you can use the ones that are available on your operating  
> > > > > system like vmstat or dstat.
> > > > > 
> > > > > --  
> > > > > Regards,  
> > > > > Rafał Kuć  
> > > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > > > Elasticsearch
> > > > > 
> > > > > But my question is still open, how to investigate such problems ? 🙂
> > > > > 
> > > > > Best regards.
> > > > > 
> > > > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > > No, I will check this, thanks.
> > > > > 
> > > > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > > > Have you looked at the Leap second problem?
> > > > > 
> > > > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > > > 
> > > > > ## David
> > > > > 
> > > > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > 
> > > > > > a écrit :
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > I have the following problem. I have 2 nodes. I started importing data  
> > > > > into one of them (node A). After few hours of importing process the node B  
> > > > > started using 100% processor and hardly answered other search and insert  
> > > > > requests. It imported 11GB of data. How can I investigate such problems?  
> > > > > Are there any logs that can help with such situation? What should I do ?
> > > > > 
> > > > > Thank you for help.
> > > > > 
> > > > > ## Best regards. Marcin Dojwa.
> > > > > 
> > > > > --
> > > > > 
> > > > > --
> > > > > 
> > > > > --
> > > > > 
> > > > > --
> > > > > 
> > > > > --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [September 27, 2012, 11:19am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/13 "2012-09-27T11:19:35Z")

</div>

One one thing, I am not sure if it hangs on bulk indexing or on 'delete by  
query' after bulk indexing.

2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> Fix for 1.: each bulk has 10000 inserts
> 
> Best regards.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > Oh, and there is nothing in gc log.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure  
> > > if that will help.
> > > 
> > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > 
> > > > Hi,
> > > > 
> > > > I still need help in this 🙂 I have the following information:
> > > > 
> > > > 1. 2 nodes (es1 and es2)
> > > > 2. 1 index Users with 30 shards and 1 replica
> > > > 
> > > > I do the following:
> > > > 
> > > > 1. On es2 I run consecutively many bulk index operations (each bulk has  
> > > > 1000 inserts) and I do about 1000 such bulk inserts.
> > > > 2. At some point of inserting es1 uses about 300% of processor and bulk  
> > > > insert hangs. Then I stop it.
> > > > 3. After 1 day doing nothing with any of ES nodes (I turn of everything  
> > > > that uses ES) es1 uses 100% of processor and es2 is OK.  
> > > > es1:  
> > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > 
> > > > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> > > > 
> > > > es2:  
> > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > 
> > > > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java
> > > > 
> > > > 1. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > > > {
> > > > 
> > > > }  
> > > > 5. es1: vmstat 5 5  
> > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > ----cpu----  
> > > > r b swpd free buff cache si so bi bo in cs us sy  
> > > > id wa  
> > > > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1  
> > > > 0 99 0  
> > > > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16  
> > > > 0 84 0  
> > > > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17  
> > > > 0 83 0  
> > > > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16  
> > > > 0 84 0  
> > > > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18  
> > > > 0 82 0
> > > > 
> > > > 1. es2: vmstat 5 5  
> > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > ----cpu----  
> > > > r b swpd free buff cache si so bi bo in cs us sy  
> > > > id wa  
> > > > 0 0 0 11327116 253432 1299284 0 0 1 13 16 16 0  
> > > > 0 100 0  
> > > > 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154 0  
> > > > 0 100 0  
> > > > 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174 0  
> > > > 0 100 0  
> > > > 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225 0  
> > > > 0 100 0  
> > > > 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109 0  
> > > > 0 100 0
> > > > 
> > > > 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows that  
> > > > everything is OK
> > > > 
> > > > 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> > > > everything is OK and it shows that:  
> > > > OS - CPU:  
> > > > Total: 100%  
> > > > User: 15%  
> > > > Sys: 0%
> > > > 
> > > > OS - LOAD:  
> > > > 2: 1.11  
> > > > 1: 1.24  
> > > > 0: 1.24
> > > > 
> > > > Process - CPU:  
> > > > Total: 400%  
> > > > Process: 123%
> > > > 
> > > > 1. After restarting elasticsearch on es1 and stabilizing es1 works fine  
> > > > but the same situation happens on es2 (100% processor usage). The only way  
> > > > to fix that is stopping all ES nodes and starting them again.
> > > > 
> > > > Is there anything I can do to investigate this problem?
> > > > 
> > > > Thanks for help.
> > > > 
> > > > Best regards.
> > > > 
> > > > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > 
> > > > > Thank you Rafał.
> > > > > 
> > > > > Best regards  
> > > > > Marcin Dojwa
> > > > > 
> > > > > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > > > > 
> > > > > > Hello!
> > > > > > 
> > > > > > You should have the following section in your elasticsearch.yml file:
> > > > > > 
> > > > > > ################################## GC Logging  
> > > > > > ################################
> > > > > > 
> > > > > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > > > > #monitor.jvm.gc.ParNew.info: 700ms  
> > > > > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > > > > 
> > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > > > > 
> > > > > > Just remove the comment character, restart Elasticsearch and you will  
> > > > > > have that being logged.
> > > > > > 
> > > > > > _--  
> > > > > > Regards,  
> > > > > > Rafał Kuć  
> > > > > > Sematext :: \*  
> > > > > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > > > > 
> > > > > > - 
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Thank you very much for your help, I will check this. Could you tell  
> > > > > > me how to enable GC logging in ES ? Thanks.
> > > > > > 
> > > > > > Best regards.
> > > > > > 
> > > > > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > > > > Hello Marcin!
> > > > > > 
> > > > > > Did you query Elasticsearch or was it only indexing data ?
> > > > > > 
> > > > > > Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> > > > > > sure that the problems you are experiencing are not garbage collection  
> > > > > > related.
> > > > > > 
> > > > > > There are some tools that can help you diagnosing what is happening  
> > > > > > with your nodes - look at the front ends mentioned on Elasticsearch  
> > > > > > clients page (  
> > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > > > > > Of course you can use the ones that are available on your operating  
> > > > > > system like vmstat or dstat.
> > > > > > 
> > > > > > --  
> > > > > > Regards,  
> > > > > > Rafał Kuć  
> > > > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > > > > Elasticsearch
> > > > > > 
> > > > > > But my question is still open, how to investigate such problems ? 🙂
> > > > > > 
> > > > > > Best regards.
> > > > > > 
> > > > > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > > > No, I will check this, thanks.
> > > > > > 
> > > > > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > > > > Have you looked at the Leap second problem?
> > > > > > 
> > > > > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > > > > 
> > > > > > ## David
> > > > > > 
> > > > > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > 
> > > > > > > a écrit :
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > I have the following problem. I have 2 nodes. I started importing  
> > > > > > data into one of them (node A). After few hours of importing process the  
> > > > > > node B started using 100% processor and hardly answered other search and  
> > > > > > insert requests. It imported 11GB of data. How can I investigate such  
> > > > > > problems? Are there any logs that can help with such situation? What should  
> > > > > > I do ?
> > > > > > 
> > > > > > Thank you for help.
> > > > > > 
> > > > > > ## Best regards. Marcin Dojwa.
> > > > > > 
> > > > > > --
> > > > > > 
> > > > > > --
> > > > > > 
> > > > > > --
> > > > > > 
> > > > > > --
> > > > > > 
> > > > > > --

--

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [September 28, 2012, 1:01pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/14 "2012-09-28T13:01:41Z")

</div>

Can you issue: curl localhost:9200/\_nodes/hot\_threads and gist the response?

On Sep 27, 2012, at 1:19 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:

> One one thing, I am not sure if it hangs on bulk indexing or on 'delete by query' after bulk indexing.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> Fix for 1.: each bulk has 10000 inserts
> 
> Best regards.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> Oh, and there is nothing in gc log.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure if that will help.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> Hi,
> 
> I still need help in this 🙂 I have the following information:
> 
> 1. 2 nodes (es1 and es2)
> 2. 1 index Users with 30 shards and 1 replica
> 
> I do the following:
> 
> 1. On es2 I run consecutively many bulk index operations (each bulk has 1000 inserts) and I do about 1000 such bulk inserts.
> 2. At some point of inserting es1 uses about 300% of processor and bulk insert hangs. Then I stop it.
> 3. After 1 day doing nothing with any of ES nodes (I turn of everything that uses ES) es1 uses 100% of processor and es2 is OK.  
> es1:  
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  
> 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java  
> es2:  
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  
> 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java
> 4. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> {
> 
> cluster\_name: "production",
> 
> status: "green",
> 
> timed\_out: false,
> 
> number\_of\_nodes: 2,
> 
> number\_of\_data\_nodes: 2,
> 
> active\_primary\_shards: 30,
> 
> active\_shards: 60,
> 
> relocating\_shards: 0,
> 
> initializing\_shards: 0,
> 
> unassigned\_shards: 0
> 
> }  
> 5. es1: vmstat 5 5
> 
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----  
> r b swpd free buff cache si so bi bo in cs us sy id wa  
> 1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1 0 99 0
> 
> 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16 0 84 0  
> 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17 0 83 0  
> 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16 0 84 0
> 
> 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18 0 82 0
> 
> 1. es2: vmstat 5 5  
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----  
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 
> 0 0 0 11327116 253432 1299284 0 0 1 13 16 16 0 0 100 0  
> 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154 0 0 100 0  
> 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174 0 0 100 0
> 
> 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225 0 0 100 0  
> 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109 0 0 100 0
> 
> 1. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows that everything is OK
> 
> 2. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that everything is OK and it shows that:  
> OS - CPU:  
> Total: 100%  
> User: 15%  
> Sys: 0%
> 
> OS - LOAD:  
> 2: 1.11  
> 1: 1.24  
> 0: 1.24
> 
> Process - CPU:  
> Total: 400%  
> Process: 123%
> 
> 1. After restarting elasticsearch on es1 and stabilizing es1 works fine but the same situation happens on es2 (100% processor usage). The only way to fix that is stopping all ES nodes and starting them again.
> 
> Is there anything I can do to investigate this problem?
> 
> Thanks for help.
> 
> Best regards.
> 
> 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> Thank you Rafał.
> 
> Best regards  
> Marcin Dojwa
> 
> 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> Hello!
> 
> You should have the following section in your elasticsearch.yml file:
> 
> ################################## GC Logging ################################
> 
> #monitor.jvm.gc.ParNew.warn: 1000ms  
> #monitor.jvm.gc.ParNew.info: 700ms  
> #monitor.jvm.gc.ParNew.debug: 400ms
> 
> #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> 
> Just remove the comment character, restart Elasticsearch and you will have that being logged.
> 
> --  
> Regards,  
> Rafał Kuć  
> Sematext ::  
> [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch
> 
> Hi,
> 
> Thank you very much for your help, I will check this. Could you tell me how to enable GC logging in ES ? Thanks.
> 
> Best regards.
> 
> 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> Hello Marcin!
> 
> Did you query Elasticsearch or was it only indexing data ?
> 
> Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> sure that the problems you are experiencing are not garbage collection  
> related.
> 
> There are some tools that can help you diagnosing what is happening  
> with your nodes - look at the front ends mentioned on Elasticsearch  
> clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> Of course you can use the ones that are available on your operating  
> system like vmstat or dstat.
> 
> --  
> Regards,  
> Rafał Kuć  
> Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch
> 
> But my question is still open, how to investigate such problems ? 🙂
> 
> Best regards.
> 
> 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> No, I will check this, thanks.
> 
> 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> Have you looked at the Leap second problem?
> 
> [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> 
> ## David
> 
> Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > a écrit :
> 
> Hi,
> 
> I have the following problem. I have 2 nodes. I started importing data into one of them (node A). After few hours of importing process the node B started using 100% processor and hardly answered other search and insert requests. It imported 11GB of data. How can I investigate such problems? Are there any logs that can help with such situation? What should I do ?
> 
> Thank you for help.
> 
> ## Best regards. Marcin Dojwa.
> 
> --
> 
> --
> 
> --
> 
> --
> 
> --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [September 29, 2012, 3:44pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/15 "2012-09-29T15:44:07Z")

</div>

Sure, I will try to reproduce this case on Monday. I updated ES to 0.19.9  
so I will check if this happens with this version too.

Best regards.

2012/9/28 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)

> Can you issue: curl localhost:9200/\_nodes/hot\_threads and gist the  
> response?
> 
> On Sep 27, 2012, at 1:19 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:
> 
> One one thing, I am not sure if it hangs on bulk indexing or on 'delete by  
> query' after bulk indexing.
> 
> 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > Fix for 1.: each bulk has 10000 inserts
> > 
> > Best regards.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > Oh, and there is nothing in gc log.
> > > 
> > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > 
> > > > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure  
> > > > if that will help.
> > > > 
> > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > 
> > > > > Hi,
> > > > > 
> > > > > I still need help in this 🙂 I have the following information:
> > > > > 
> > > > > 1. 2 nodes (es1 and es2)
> > > > > 2. 1 index Users with 30 shards and 1 replica
> > > > > 
> > > > > I do the following:
> > > > > 
> > > > > 1. On es2 I run consecutively many bulk index operations (each bulk  
> > > > > has 1000 inserts) and I do about 1000 such bulk inserts.
> > > > > 2. At some point of inserting es1 uses about 300% of processor and  
> > > > > bulk insert hangs. Then I stop it.
> > > > > 3. After 1 day doing nothing with any of ES nodes (I turn of  
> > > > > everything that uses ES) es1 uses 100% of processor and es2 is OK.  
> > > > > es1:  
> > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > > 
> > > > > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> > > > > 
> > > > > es2:  
> > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > > 
> > > > > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java
> > > > > 
> > > > > 1. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > > > > {
> > > > > 
> > > > > }  
> > > > > 5. es1: vmstat 5 5  
> > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > ----cpu----  
> > > > > r b swpd free buff cache si so bi bo in cs us  
> > > > > sy id wa  
> > > > > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1  
> > > > > 0 99 0  
> > > > > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16  
> > > > > 0 84 0  
> > > > > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17  
> > > > > 0 83 0  
> > > > > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16  
> > > > > 0 84 0  
> > > > > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18  
> > > > > 0 82 0
> > > > > 
> > > > > 1. es2: vmstat 5 5  
> > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > ----cpu----  
> > > > > r b swpd free buff cache si so bi bo in cs us  
> > > > > sy id wa  
> > > > > 0 0 0 11327116 253432 1299284 0 0 1 13 16 16  
> > > > > 0 0 100 0  
> > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154  
> > > > > 0 0 100 0  
> > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174  
> > > > > 0 0 100 0  
> > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225  
> > > > > 0 0 100 0  
> > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109  
> > > > > 0 0 100 0
> > > > > 
> > > > > 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows  
> > > > > that everything is OK
> > > > > 
> > > > > 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> > > > > everything is OK and it shows that:  
> > > > > OS - CPU:  
> > > > > Total: 100%  
> > > > > User: 15%  
> > > > > Sys: 0%
> > > > > 
> > > > > OS - LOAD:  
> > > > > 2: 1.11  
> > > > > 1: 1.24  
> > > > > 0: 1.24
> > > > > 
> > > > > Process - CPU:  
> > > > > Total: 400%  
> > > > > Process: 123%
> > > > > 
> > > > > 1. After restarting elasticsearch on es1 and stabilizing es1 works  
> > > > > fine but the same situation happens on es2 (100% processor usage). The only  
> > > > > way to fix that is stopping all ES nodes and starting them again.
> > > > > 
> > > > > Is there anything I can do to investigate this problem?
> > > > > 
> > > > > Thanks for help.
> > > > > 
> > > > > Best regards.
> > > > > 
> > > > > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > 
> > > > > > Thank you Rafał.
> > > > > > 
> > > > > > Best regards  
> > > > > > Marcin Dojwa
> > > > > > 
> > > > > > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > > > > > 
> > > > > > > Hello!
> > > > > > > 
> > > > > > > You should have the following section in your elasticsearch.yml file:
> > > > > > > 
> > > > > > > ################################## GC Logging  
> > > > > > > ################################
> > > > > > > 
> > > > > > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > > > > > #monitor.jvm.gc.ParNew.info [http://monitor.jvm.gc.parnew.info/](http://monitor.jvm.gc.parnew.info/):  
> > > > > > > 700ms  
> > > > > > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > > > > > 
> > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.info[http://monitor.jvm.gc.concurrentmarksweep.info/](http://monitor.jvm.gc.concurrentmarksweep.info/):  
> > > > > > > 5s  
> > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > > > > > 
> > > > > > > Just remove the comment character, restart Elasticsearch and you  
> > > > > > > will have that being logged.
> > > > > > > 
> > > > > > > _--  
> > > > > > > Regards,  
> > > > > > > Rafał Kuć  
> > > > > > > Sematext :: \*  
> > > > > > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > > > > > 
> > > > > > > - 
> > > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > Thank you very much for your help, I will check this. Could you tell  
> > > > > > > me how to enable GC logging in ES ? Thanks.
> > > > > > > 
> > > > > > > Best regards.
> > > > > > > 
> > > > > > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > > > > > Hello Marcin!
> > > > > > > 
> > > > > > > Did you query Elasticsearch or was it only indexing data ?
> > > > > > > 
> > > > > > > Anyway, first of all I would enable GC logging in Elasticsearch, to  
> > > > > > > be  
> > > > > > > sure that the problems you are experiencing are not garbage  
> > > > > > > collection  
> > > > > > > related.
> > > > > > > 
> > > > > > > There are some tools that can help you diagnosing what is happening  
> > > > > > > with your nodes - look at the front ends mentioned on Elasticsearch  
> > > > > > > clients page (  
> > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > > > > > > Of course you can use the ones that are available on your operating  
> > > > > > > system like vmstat or dstat.
> > > > > > > 
> > > > > > > --  
> > > > > > > Regards,  
> > > > > > > Rafał Kuć  
> > > > > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > > > > > Elasticsearch
> > > > > > > 
> > > > > > > But my question is still open, how to investigate such problems ? 🙂
> > > > > > > 
> > > > > > > Best regards.
> > > > > > > 
> > > > > > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > > > > No, I will check this, thanks.
> > > > > > > 
> > > > > > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > > > > > Have you looked at the Leap second problem?
> > > > > > > 
> > > > > > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > > > > > 
> > > > > > > ## David
> > > > > > > 
> > > > > > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > > 
> > > > > > > > a écrit :
> > > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I have the following problem. I have 2 nodes. I started importing  
> > > > > > > data into one of them (node A). After few hours of importing process the  
> > > > > > > node B started using 100% processor and hardly answered other search and  
> > > > > > > insert requests. It imported 11GB of data. How can I investigate such  
> > > > > > > problems? Are there any logs that can help with such situation? What should  
> > > > > > > I do ?
> > > > > > > 
> > > > > > > Thank you for help.
> > > > > > > 
> > > > > > > ## Best regards. Marcin Dojwa.
> > > > > > > 
> > > > > > > --
> > > > > > > 
> > > > > > > --
> > > > > > > 
> > > > > > > --
> > > > > > > 
> > > > > > > --
> > > > > > > 
> > > > > > > --
> 
> --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [October 2, 2012, 8:16am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/16 "2012-10-02T08:16:34Z")

</div>

Hi,

I attached 4 different (I think) snapshots of curl  
localhost:9200/\_nodes/hot\_threads here:

> <https://gist.github.com/anonymous/f2826104eb5b288d3fb0>
>
> There are more than three files. show original

Best regards.  
Marcin Dojwa

2012/9/29 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)

> Sure, I will try to reproduce this case on Monday. I updated ES to 0.19.9  
> so I will check if this happens with this version too.
> 
> Best regards.
> 
> 2012/9/28 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> 
> > Can you issue: curl localhost:9200/\_nodes/hot\_threads and gist the  
> > response?
> > 
> > On Sep 27, 2012, at 1:19 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > wrote:
> > 
> > One one thing, I am not sure if it hangs on bulk indexing or on 'delete  
> > by query' after bulk indexing.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > Fix for 1.: each bulk has 10000 inserts
> > > 
> > > Best regards.
> > > 
> > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > 
> > > > Oh, and there is nothing in gc log.
> > > > 
> > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > 
> > > > > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not  
> > > > > sure if that will help.
> > > > > 
> > > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > I still need help in this 🙂 I have the following information:
> > > > > > 
> > > > > > 1. 2 nodes (es1 and es2)
> > > > > > 2. 1 index Users with 30 shards and 1 replica
> > > > > > 
> > > > > > I do the following:
> > > > > > 
> > > > > > 1. On es2 I run consecutively many bulk index operations (each bulk  
> > > > > > has 1000 inserts) and I do about 1000 such bulk inserts.
> > > > > > 2. At some point of inserting es1 uses about 300% of processor and  
> > > > > > bulk insert hangs. Then I stop it.
> > > > > > 3. After 1 day doing nothing with any of ES nodes (I turn of  
> > > > > > everything that uses ES) es1 uses 100% of processor and es2 is OK.  
> > > > > > es1:  
> > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > > > 
> > > > > > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> > > > > > 
> > > > > > es2:  
> > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > > > 
> > > > > > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java
> > > > > > 
> > > > > > 1. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > > > > > {
> > > > > > 
> > > > > > }  
> > > > > > 5. es1: vmstat 5 5  
> > > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > > ----cpu----  
> > > > > > r b swpd free buff cache si so bi bo in cs us  
> > > > > > sy id wa  
> > > > > > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1  
> > > > > > 1 0 99 0  
> > > > > > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559  
> > > > > > 16 0 84 0  
> > > > > > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584  
> > > > > > 17 0 83 0  
> > > > > > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552  
> > > > > > 16 0 84 0  
> > > > > > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604  
> > > > > > 18 0 82 0
> > > > > > 
> > > > > > 1. es2: vmstat 5 5  
> > > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > > ----cpu----  
> > > > > > r b swpd free buff cache si so bi bo in cs us  
> > > > > > sy id wa  
> > > > > > 0 0 0 11327116 253432 1299284 0 0 1 13 16 16  
> > > > > > 0 0 100 0  
> > > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154  
> > > > > > 0 0 100 0  
> > > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174  
> > > > > > 0 0 100 0  
> > > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225  
> > > > > > 0 0 100 0  
> > > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109  
> > > > > > 0 0 100 0
> > > > > > 
> > > > > > 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows  
> > > > > > that everything is OK
> > > > > > 
> > > > > > 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> > > > > > everything is OK and it shows that:  
> > > > > > OS - CPU:  
> > > > > > Total: 100%  
> > > > > > User: 15%  
> > > > > > Sys: 0%
> > > > > > 
> > > > > > OS - LOAD:  
> > > > > > 2: 1.11  
> > > > > > 1: 1.24  
> > > > > > 0: 1.24
> > > > > > 
> > > > > > Process - CPU:  
> > > > > > Total: 400%  
> > > > > > Process: 123%
> > > > > > 
> > > > > > 1. After restarting elasticsearch on es1 and stabilizing es1 works  
> > > > > > fine but the same situation happens on es2 (100% processor usage). The only  
> > > > > > way to fix that is stopping all ES nodes and starting them again.
> > > > > > 
> > > > > > Is there anything I can do to investigate this problem?
> > > > > > 
> > > > > > Thanks for help.
> > > > > > 
> > > > > > Best regards.
> > > > > > 
> > > > > > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > 
> > > > > > > Thank you Rafał.
> > > > > > > 
> > > > > > > Best regards  
> > > > > > > Marcin Dojwa
> > > > > > > 
> > > > > > > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > > > > > > 
> > > > > > > > Hello!
> > > > > > > > 
> > > > > > > > You should have the following section in your elasticsearch.yml  
> > > > > > > > file:
> > > > > > > > 
> > > > > > > > ################################## GC Logging  
> > > > > > > > ################################
> > > > > > > > 
> > > > > > > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > > > > > > #monitor.jvm.gc.ParNew.info [http://monitor.jvm.gc.parnew.info/](http://monitor.jvm.gc.parnew.info/):  
> > > > > > > > 700ms  
> > > > > > > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > > > > > > 
> > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.info[http://monitor.jvm.gc.concurrentmarksweep.info/](http://monitor.jvm.gc.concurrentmarksweep.info/):  
> > > > > > > > 5s  
> > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > > > > > > 
> > > > > > > > Just remove the comment character, restart Elasticsearch and you  
> > > > > > > > will have that being logged.
> > > > > > > > 
> > > > > > > > _--  
> > > > > > > > Regards,  
> > > > > > > > Rafał Kuć  
> > > > > > > > Sematext :: \*  
> > > > > > > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > > > > > > 
> > > > > > > > - 
> > > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > Thank you very much for your help, I will check this. Could you  
> > > > > > > > tell me how to enable GC logging in ES ? Thanks.
> > > > > > > > 
> > > > > > > > Best regards.
> > > > > > > > 
> > > > > > > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > > > > > > Hello Marcin!
> > > > > > > > 
> > > > > > > > Did you query Elasticsearch or was it only indexing data ?
> > > > > > > > 
> > > > > > > > Anyway, first of all I would enable GC logging in Elasticsearch, to  
> > > > > > > > be  
> > > > > > > > sure that the problems you are experiencing are not garbage  
> > > > > > > > collection  
> > > > > > > > related.
> > > > > > > > 
> > > > > > > > There are some tools that can help you diagnosing what is happening  
> > > > > > > > with your nodes - look at the front ends mentioned on Elasticsearch  
> > > > > > > > clients page (  
> > > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > > > > > > > Of course you can use the ones that are available on your operating  
> > > > > > > > system like vmstat or dstat.
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > Regards,  
> > > > > > > > Rafał Kuć  
> > > > > > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > > > > > > Elasticsearch
> > > > > > > > 
> > > > > > > > But my question is still open, how to investigate such problems ? 🙂
> > > > > > > > 
> > > > > > > > Best regards.
> > > > > > > > 
> > > > > > > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > > > > > No, I will check this, thanks.
> > > > > > > > 
> > > > > > > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > > > > > > Have you looked at the Leap second problem?
> > > > > > > > 
> > > > > > > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > > > > > > 
> > > > > > > > ## David
> > > > > > > > 
> > > > > > > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > > > 
> > > > > > > > > a écrit :
> > > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > I have the following problem. I have 2 nodes. I started importing  
> > > > > > > > data into one of them (node A). After few hours of importing process the  
> > > > > > > > node B started using 100% processor and hardly answered other search and  
> > > > > > > > insert requests. It imported 11GB of data. How can I investigate such  
> > > > > > > > problems? Are there any logs that can help with such situation? What should  
> > > > > > > > I do ?
> > > > > > > > 
> > > > > > > > Thank you for help.
> > > > > > > > 
> > > > > > > > ## Best regards. Marcin Dojwa.
> > > > > > > > 
> > > > > > > > --
> > > > > > > > 
> > > > > > > > --
> > > > > > > > 
> > > > > > > > --
> > > > > > > > 
> > > > > > > > --
> > > > > > > > 
> > > > > > > > --
> > 
> > --
> > 
> > --

--

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [October 2, 2012, 6:35pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/17 "2012-10-02T18:35:56Z")

</div>

It seems like its doing the delete by query with nested documents. Can you share your mapping, and sample data that you index, and the delete\_by\_query that you are using?

On Oct 2, 2012, at 4:16 AM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:

> Hi,
> 
> I attached 4 different (I think) snapshots of curl localhost:9200/\_nodes/hot\_threads here: [ES 100% processor usage · GitHub](https://gist.github.com/f2826104eb5b288d3fb0)
> 
> Best regards.  
> Marcin Dojwa
> 
> 2012/9/29 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> Sure, I will try to reproduce this case on Monday. I updated ES to 0.19.9 so I will check if this happens with this version too.
> 
> Best regards.
> 
> 2012/9/28 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)  
> Can you issue: curl localhost:9200/\_nodes/hot\_threads and gist the response?
> 
> On Sep 27, 2012, at 1:19 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:
> 
> > One one thing, I am not sure if it hangs on bulk indexing or on 'delete by query' after bulk indexing.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > Fix for 1.: each bulk has 10000 inserts
> > 
> > Best regards.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > Oh, and there is nothing in gc log.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure if that will help.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > Hi,
> > 
> > I still need help in this 🙂 I have the following information:
> > 
> > 1. 2 nodes (es1 and es2)
> > 2. 1 index Users with 30 shards and 1 replica
> > 
> > I do the following:
> > 
> > 1. On es2 I run consecutively many bulk index operations (each bulk has 1000 inserts) and I do about 1000 such bulk inserts.
> > 2. At some point of inserting es1 uses about 300% of processor and bulk insert hangs. Then I stop it.
> > 3. After 1 day doing nothing with any of ES nodes (I turn of everything that uses ES) es1 uses 100% of processor and es2 is OK.  
> > es1:  
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  
> > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java  
> > es2:  
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  
> > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java
> > 4. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > {
> > 
> > cluster\_name: "production",
> > 
> > status: "green",
> > 
> > timed\_out: false,
> > 
> > number\_of\_nodes: 2,
> > 
> > number\_of\_data\_nodes: 2,
> > 
> > active\_primary\_shards: 30,
> > 
> > active\_shards: 60,
> > 
> > relocating\_shards: 0,
> > 
> > initializing\_shards: 0,
> > 
> > unassigned\_shards: 0
> > 
> > }  
> > 5. es1: vmstat 5 5
> > 
> > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----  
> > r b swpd free buff cache si so bi bo in cs us sy id wa  
> > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1 1 0 99 0
> > 
> > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559 16 0 84 0  
> > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584 17 0 83 0  
> > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552 16 0 84 0
> > 
> > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604 18 0 82 0
> > 
> > 1. es2: vmstat 5 5  
> > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----  
> > r b swpd free buff cache si so bi bo in cs us sy id wa
> > 
> > 0 0 0 11327116 253432 1299284 0 0 1 13 16 16 0 0 100 0  
> > 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154 0 0 100 0  
> > 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174 0 0 100 0
> > 
> > 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225 0 0 100 0  
> > 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109 0 0 100 0
> > 
> > 1. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows that everything is OK
> > 
> > 2. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that everything is OK and it shows that:  
> > OS - CPU:  
> > Total: 100%  
> > User: 15%  
> > Sys: 0%
> > 
> > OS - LOAD:  
> > 2: 1.11  
> > 1: 1.24  
> > 0: 1.24
> > 
> > Process - CPU:  
> > Total: 400%  
> > Process: 123%
> > 
> > 1. After restarting elasticsearch on es1 and stabilizing es1 works fine but the same situation happens on es2 (100% processor usage). The only way to fix that is stopping all ES nodes and starting them again.
> > 
> > Is there anything I can do to investigate this problem?
> > 
> > Thanks for help.
> > 
> > Best regards.
> > 
> > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > Thank you Rafał.
> > 
> > Best regards  
> > Marcin Dojwa
> > 
> > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > Hello!
> > 
> > You should have the following section in your elasticsearch.yml file:
> > 
> > ################################## GC Logging ################################
> > 
> > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > #monitor.jvm.gc.ParNew.info: 700ms  
> > #monitor.jvm.gc.ParNew.debug: 400ms
> > 
> > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > #monitor.jvm.gc.ConcurrentMarkSweep.info: 5s  
> > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > 
> > Just remove the comment character, restart Elasticsearch and you will have that being logged.
> > 
> > --  
> > Regards,  
> > Rafał Kuć  
> > Sematext ::  
> > [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch
> > 
> > Hi,
> > 
> > Thank you very much for your help, I will check this. Could you tell me how to enable GC logging in ES ? Thanks.
> > 
> > Best regards.
> > 
> > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > Hello Marcin!
> > 
> > Did you query Elasticsearch or was it only indexing data ?
> > 
> > Anyway, first of all I would enable GC logging in Elasticsearch, to be  
> > sure that the problems you are experiencing are not garbage collection  
> > related.
> > 
> > There are some tools that can help you diagnosing what is happening  
> > with your nodes - look at the front ends mentioned on Elasticsearch  
> > clients page ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > Of course you can use the ones that are available on your operating  
> > system like vmstat or dstat.
> > 
> > --  
> > Regards,  
> > Rafał Kuć  
> > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch - Elasticsearch
> > 
> > But my question is still open, how to investigate such problems ? 🙂
> > 
> > Best regards.
> > 
> > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > No, I will check this, thanks.
> > 
> > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > Have you looked at the Leap second problem?
> > 
> > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > 
> > ## David
> > 
> > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > a écrit :
> > 
> > Hi,
> > 
> > I have the following problem. I have 2 nodes. I started importing data into one of them (node A). After few hours of importing process the node B started using 100% processor and hardly answered other search and insert requests. It imported 11GB of data. How can I investigate such problems? Are there any logs that can help with such situation? What should I do ?
> > 
> > Thank you for help.
> > 
> > ## Best regards. Marcin Dojwa.
> > 
> > --
> > 
> > --
> > 
> > --
> > 
> > --
> > 
> > --
> > 
> > --
> 
> --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [October 3, 2012, 8:50am UTC](https://discuss.elastic.co/t/100-processor-usage/8749/18 "2012-10-03T08:50:47Z")

</div>

Hi,

The exact mapping is here: [ES 100% processor usage · GitHub](https://gist.github.com/bbde44039b20a57eebf0)  
I guess that this is about chat document type because this is the only one  
with nested mapping. I can not give you exact data that could possibly  
cause such a situation because this is our clients' private data. Are you  
able to try reproduce the problem with this information about mapping only?  
If not I will try to prepare fake example data.

Best regards.

2012/10/2 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)

> It seems like its doing the delete by query with nested documents. Can you  
> share your mapping, and sample data that you index, and the delete\_by\_query  
> that you are using?
> 
> On Oct 2, 2012, at 4:16 AM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:
> 
> Hi,
> 
> I attached 4 different (I think) snapshots of curl  
> localhost:9200/\_nodes/hot\_threads here:  
> [ES 100% processor usage · GitHub](https://gist.github.com/f2826104eb5b288d3fb0)
> 
> Best regards.  
> Marcin Dojwa
> 
> 2012/9/29 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> 
> > Sure, I will try to reproduce this case on Monday. I updated ES to 0.19.9  
> > so I will check if this happens with this version too.
> > 
> > Best regards.
> > 
> > 2012/9/28 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> > 
> > > Can you issue: curl localhost:9200/\_nodes/hot\_threads and gist the  
> > > response?
> > > 
> > > On Sep 27, 2012, at 1:19 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > wrote:
> > > 
> > > One one thing, I am not sure if it hangs on bulk indexing or on 'delete  
> > > by query' after bulk indexing.
> > > 
> > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > 
> > > > Fix for 1.: each bulk has 10000 inserts
> > > > 
> > > > Best regards.
> > > > 
> > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > 
> > > > > Oh, and there is nothing in gc log.
> > > > > 
> > > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > 
> > > > > > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not  
> > > > > > sure if that will help.
> > > > > > 
> > > > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I still need help in this 🙂 I have the following information:
> > > > > > > 
> > > > > > > 1. 2 nodes (es1 and es2)
> > > > > > > 2. 1 index Users with 30 shards and 1 replica
> > > > > > > 
> > > > > > > I do the following:
> > > > > > > 
> > > > > > > 1. On es2 I run consecutively many bulk index operations (each bulk  
> > > > > > > has 1000 inserts) and I do about 1000 such bulk inserts.
> > > > > > > 2. At some point of inserting es1 uses about 300% of processor and  
> > > > > > > bulk insert hangs. Then I stop it.
> > > > > > > 3. After 1 day doing nothing with any of ES nodes (I turn of  
> > > > > > > everything that uses ES) es1 uses 100% of processor and es2 is OK.  
> > > > > > > es1:  
> > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > > > > 
> > > > > > > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> > > > > > > 
> > > > > > > es2:  
> > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > > > > 
> > > > > > > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java
> > > > > > > 
> > > > > > > 1. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > > > > > > {
> > > > > > > 
> > > > > > > }  
> > > > > > > 5. es1: vmstat 5 5  
> > > > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > > > ----cpu----  
> > > > > > > r b swpd free buff cache si so bi bo in cs us  
> > > > > > > sy id wa  
> > > > > > > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1  
> > > > > > > 1 0 99 0  
> > > > > > > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559  
> > > > > > > 16 0 84 0  
> > > > > > > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584  
> > > > > > > 17 0 83 0  
> > > > > > > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552  
> > > > > > > 16 0 84 0  
> > > > > > > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604  
> > > > > > > 18 0 82 0
> > > > > > > 
> > > > > > > 1. es2: vmstat 5 5  
> > > > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > > > ----cpu----  
> > > > > > > r b swpd free buff cache si so bi bo in cs us  
> > > > > > > sy id wa  
> > > > > > > 0 0 0 11327116 253432 1299284 0 0 1 13 16 16  
> > > > > > > 0 0 100 0  
> > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 897 2154  
> > > > > > > 0 0 100 0  
> > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 914 2174  
> > > > > > > 0 0 100 0  
> > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 1057 2225  
> > > > > > > 0 0 100 0  
> > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 874 2109  
> > > > > > > 0 0 100 0
> > > > > > > 
> > > > > > > 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows  
> > > > > > > that everything is OK
> > > > > > > 
> > > > > > > 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> > > > > > > everything is OK and it shows that:  
> > > > > > > OS - CPU:  
> > > > > > > Total: 100%  
> > > > > > > User: 15%  
> > > > > > > Sys: 0%
> > > > > > > 
> > > > > > > OS - LOAD:  
> > > > > > > 2: 1.11  
> > > > > > > 1: 1.24  
> > > > > > > 0: 1.24
> > > > > > > 
> > > > > > > Process - CPU:  
> > > > > > > Total: 400%  
> > > > > > > Process: 123%
> > > > > > > 
> > > > > > > 1. After restarting elasticsearch on es1 and stabilizing es1 works  
> > > > > > > fine but the same situation happens on es2 (100% processor usage). The only  
> > > > > > > way to fix that is stopping all ES nodes and starting them again.
> > > > > > > 
> > > > > > > Is there anything I can do to investigate this problem?
> > > > > > > 
> > > > > > > Thanks for help.
> > > > > > > 
> > > > > > > Best regards.
> > > > > > > 
> > > > > > > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > > 
> > > > > > > > Thank you Rafał.
> > > > > > > > 
> > > > > > > > Best regards  
> > > > > > > > Marcin Dojwa
> > > > > > > > 
> > > > > > > > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > > > > > > > 
> > > > > > > > > Hello!
> > > > > > > > > 
> > > > > > > > > You should have the following section in your elasticsearch.yml  
> > > > > > > > > file:
> > > > > > > > > 
> > > > > > > > > ################################## GC Logging  
> > > > > > > > > ################################
> > > > > > > > > 
> > > > > > > > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > > > > > > > #monitor.jvm.gc.ParNew.info [http://monitor.jvm.gc.parnew.info/](http://monitor.jvm.gc.parnew.info/):  
> > > > > > > > > 700ms  
> > > > > > > > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > > > > > > > 
> > > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.info[http://monitor.jvm.gc.concurrentmarksweep.info/](http://monitor.jvm.gc.concurrentmarksweep.info/):  
> > > > > > > > > 5s  
> > > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > > > > > > > 
> > > > > > > > > Just remove the comment character, restart Elasticsearch and you  
> > > > > > > > > will have that being logged.
> > > > > > > > > 
> > > > > > > > > _--  
> > > > > > > > > Regards,  
> > > > > > > > > Rafał Kuć  
> > > > > > > > > Sematext :: \*  
> > > > > > > > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > > > > > > > 
> > > > > > > > > - 
> > > > > > > > > 
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > Thank you very much for your help, I will check this. Could you  
> > > > > > > > > tell me how to enable GC logging in ES ? Thanks.
> > > > > > > > > 
> > > > > > > > > Best regards.
> > > > > > > > > 
> > > > > > > > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > > > > > > > Hello Marcin!
> > > > > > > > > 
> > > > > > > > > Did you query Elasticsearch or was it only indexing data ?
> > > > > > > > > 
> > > > > > > > > Anyway, first of all I would enable GC logging in Elasticsearch,  
> > > > > > > > > to be  
> > > > > > > > > sure that the problems you are experiencing are not garbage  
> > > > > > > > > collection  
> > > > > > > > > related.
> > > > > > > > > 
> > > > > > > > > There are some tools that can help you diagnosing what is happening  
> > > > > > > > > with your nodes - look at the front ends mentioned on Elasticsearch  
> > > > > > > > > clients page (  
> > > > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > > > > > > > > Of course you can use the ones that are available on your operating  
> > > > > > > > > system like vmstat or dstat.
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > Regards,  
> > > > > > > > > Rafał Kuć  
> > > > > > > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > > > > > > > Elasticsearch
> > > > > > > > > 
> > > > > > > > > But my question is still open, how to investigate such problems ?  
> > > > > > > > > 🙂
> > > > > > > > > 
> > > > > > > > > Best regards.
> > > > > > > > > 
> > > > > > > > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > > > > > > No, I will check this, thanks.
> > > > > > > > > 
> > > > > > > > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > > > > > > > Have you looked at the Leap second problem?
> > > > > > > > > 
> > > > > > > > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > > > > > > > 
> > > > > > > > > ## David
> > > > > > > > > 
> > > > > > > > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > > > > 
> > > > > > > > > > a écrit :
> > > > > > > > > 
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > I have the following problem. I have 2 nodes. I started importing  
> > > > > > > > > data into one of them (node A). After few hours of importing process the  
> > > > > > > > > node B started using 100% processor and hardly answered other search and  
> > > > > > > > > insert requests. It imported 11GB of data. How can I investigate such  
> > > > > > > > > problems? Are there any logs that can help with such situation? What should  
> > > > > > > > > I do ?
> > > > > > > > > 
> > > > > > > > > Thank you for help.
> > > > > > > > > 
> > > > > > > > > ## Best regards. Marcin Dojwa.
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > 
> > > > > > > > > --
> > > 
> > > --
> > > 
> > > --
> 
> --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\
**Post date:** [October 3, 2012, 3:05pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/19 "2012-10-03T15:05:30Z")

</div>

Hi Marcin,

Are you using one or more aliases when executing the delete by query?

Martijn

On 3 October 2012 10:50, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:

> Hi,
> 
> The exact mapping is here: [ES 100% processor usage · GitHub](https://gist.github.com/bbde44039b20a57eebf0)  
> I guess that this is about chat document type because this is the only one  
> with nested mapping. I can not give you exact data that could possibly  
> cause such a situation because this is our clients' private data. Are you  
> able to try reproduce the problem with this information about mapping only?  
> If not I will try to prepare fake example data.
> 
> Best regards.
> 
> 2012/10/2 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> 
> > It seems like its doing the delete by query with nested documents. Can  
> > you share your mapping, and sample data that you index, and the  
> > delete\_by\_query that you are using?
> > 
> > On Oct 2, 2012, at 4:16 AM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:
> > 
> > Hi,
> > 
> > I attached 4 different (I think) snapshots of curl  
> > localhost:9200/\_nodes/hot\_threads here:  
> > [ES 100% processor usage · GitHub](https://gist.github.com/f2826104eb5b288d3fb0)
> > 
> > Best regards.  
> > Marcin Dojwa
> > 
> > 2012/9/29 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > > Sure, I will try to reproduce this case on Monday. I updated ES to  
> > > 0.19.9 so I will check if this happens with this version too.
> > > 
> > > Best regards.
> > > 
> > > 2012/9/28 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> > > 
> > > > Can you issue: curl localhost:9200/\_nodes/hot\_threads and gist the  
> > > > response?
> > > > 
> > > > On Sep 27, 2012, at 1:19 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > wrote:
> > > > 
> > > > One one thing, I am not sure if it hangs on bulk indexing or on 'delete  
> > > > by query' after bulk indexing.
> > > > 
> > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > 
> > > > > Fix for 1.: each bulk has 10000 inserts
> > > > > 
> > > > > Best regards.
> > > > > 
> > > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > 
> > > > > > Oh, and there is nothing in gc log.
> > > > > > 
> > > > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > 
> > > > > > > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not  
> > > > > > > sure if that will help.
> > > > > > > 
> > > > > > > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > I still need help in this 🙂 I have the following information:
> > > > > > > > 
> > > > > > > > 1. 2 nodes (es1 and es2)
> > > > > > > > 2. 1 index Users with 30 shards and 1 replica
> > > > > > > > 
> > > > > > > > I do the following:
> > > > > > > > 
> > > > > > > > 1. On es2 I run consecutively many bulk index operations (each bulk  
> > > > > > > > has 1000 inserts) and I do about 1000 such bulk inserts.
> > > > > > > > 2. At some point of inserting es1 uses about 300% of processor and  
> > > > > > > > bulk insert hangs. Then I stop it.
> > > > > > > > 3. After 1 day doing nothing with any of ES nodes (I turn of  
> > > > > > > > everything that uses ES) es1 uses 100% of processor and es2 is OK.  
> > > > > > > > es1:  
> > > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > > > > > > 
> > > > > > > > 12199 es 20 0 8847m 1.8g 11m S 102 11.7 3421:26 java
> > > > > > > > 
> > > > > > > > es2:  
> > > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+  
> > > > > > > > COMMAND
> > > > > > > > 
> > > > > > > > 9816 es 20 0 8771m 2.2g 11m S 1 14.4 116:41.97 java
> > > > > > > > 
> > > > > > > > 1. [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)  
> > > > > > > > {
> > > > > > > > 
> > > > > > > > }  
> > > > > > > > 5. es1: vmstat 5 5  
> > > > > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > > > > ----cpu----  
> > > > > > > > r b swpd free buff cache si so bi bo in cs  
> > > > > > > > us sy id wa  
> > > > > > > > 1 0 0 4389668 294084 8416356 0 0 0 2 0 1  
> > > > > > > > 1 0 99 0  
> > > > > > > > 1 0 0 4389420 294084 8416356 0 0 0 0 1066 2559  
> > > > > > > > 16 0 84 0  
> > > > > > > > 1 0 0 4387312 294084 8416356 0 0 0 10 1083 2584  
> > > > > > > > 17 0 83 0  
> > > > > > > > 1 0 0 4386584 294084 8416356 0 0 0 0 1087 2552  
> > > > > > > > 16 0 84 0  
> > > > > > > > 1 0 0 4385096 294084 8416356 0 0 0 6 1102 2604  
> > > > > > > > 18 0 82 0
> > > > > > > > 
> > > > > > > > 1. es2: vmstat 5 5  
> > > > > > > > procs -----------memory---------- ---swap-- -----io---- -system--  
> > > > > > > > ----cpu----  
> > > > > > > > r b swpd free buff cache si so bi bo in cs  
> > > > > > > > us sy id wa  
> > > > > > > > 0 0 0 11327116 253432 1299284 0 0 1 13 16  
> > > > > > > > 16 0 0 100 0  
> > > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 897  
> > > > > > > > 2154 0 0 100 0  
> > > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 914  
> > > > > > > > 2174 0 0 100 0  
> > > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 0 1057  
> > > > > > > > 2225 0 0 100 0  
> > > > > > > > 0 0 0 11326984 253432 1299284 0 0 0 7 874  
> > > > > > > > 2109 0 0 100 0
> > > > > > > > 
> > > > > > > > 2. [GitHub - mobz/elasticsearch-head: A web front end for an elastic search cluster](https://github.com/mobz/elasticsearch-head) - this plugin shows  
> > > > > > > > that everything is OK
> > > > > > > > 
> > > > > > > > 3. [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) - this plugin shows that  
> > > > > > > > everything is OK and it shows that:  
> > > > > > > > OS - CPU:  
> > > > > > > > Total: 100%  
> > > > > > > > User: 15%  
> > > > > > > > Sys: 0%
> > > > > > > > 
> > > > > > > > OS - LOAD:  
> > > > > > > > 2: 1.11  
> > > > > > > > 1: 1.24  
> > > > > > > > 0: 1.24
> > > > > > > > 
> > > > > > > > Process - CPU:  
> > > > > > > > Total: 400%  
> > > > > > > > Process: 123%
> > > > > > > > 
> > > > > > > > 1. After restarting elasticsearch on es1 and stabilizing es1 works  
> > > > > > > > fine but the same situation happens on es2 (100% processor usage). The only  
> > > > > > > > way to fix that is stopping all ES nodes and starting them again.
> > > > > > > > 
> > > > > > > > Is there anything I can do to investigate this problem?
> > > > > > > > 
> > > > > > > > Thanks for help.
> > > > > > > > 
> > > > > > > > Best regards.
> > > > > > > > 
> > > > > > > > 2012/8/16 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > > > 
> > > > > > > > > Thank you Rafał.
> > > > > > > > > 
> > > > > > > > > Best regards  
> > > > > > > > > Marcin Dojwa
> > > > > > > > > 
> > > > > > > > > 2012/8/16 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)
> > > > > > > > > 
> > > > > > > > > > Hello!
> > > > > > > > > > 
> > > > > > > > > > You should have the following section in your elasticsearch.yml  
> > > > > > > > > > file:
> > > > > > > > > > 
> > > > > > > > > > ################################## GC Logging  
> > > > > > > > > > ################################
> > > > > > > > > > 
> > > > > > > > > > #monitor.jvm.gc.ParNew.warn: 1000ms  
> > > > > > > > > > #monitor.jvm.gc.ParNew.info [http://monitor.jvm.gc.parnew.info/](http://monitor.jvm.gc.parnew.info/):  
> > > > > > > > > > 700ms  
> > > > > > > > > > #monitor.jvm.gc.ParNew.debug: 400ms
> > > > > > > > > > 
> > > > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s  
> > > > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.info[http://monitor.jvm.gc.concurrentmarksweep.info/](http://monitor.jvm.gc.concurrentmarksweep.info/):  
> > > > > > > > > > 5s  
> > > > > > > > > > #monitor.jvm.gc.ConcurrentMarkSweep.debug: 2s
> > > > > > > > > > 
> > > > > > > > > > Just remove the comment character, restart Elasticsearch and you  
> > > > > > > > > > will have that being logged.
> > > > > > > > > > 
> > > > > > > > > > _--  
> > > > > > > > > > Regards,  
> > > > > > > > > > Rafał Kuć  
> > > > > > > > > > Sematext :: \*  
> > > > > > > > > > [http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - Elasticsearch
> > > > > > > > > > 
> > > > > > > > > > - 
> > > > > > > > > > 
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > Thank you very much for your help, I will check this. Could you  
> > > > > > > > > > tell me how to enable GC logging in ES ? Thanks.
> > > > > > > > > > 
> > > > > > > > > > Best regards.
> > > > > > > > > > 
> > > > > > > > > > 2012/8/15 Rafał Kuć [r.kuc@solr.pl](mailto:r.kuc@solr.pl)  
> > > > > > > > > > Hello Marcin!
> > > > > > > > > > 
> > > > > > > > > > Did you query Elasticsearch or was it only indexing data ?
> > > > > > > > > > 
> > > > > > > > > > Anyway, first of all I would enable GC logging in Elasticsearch,  
> > > > > > > > > > to be  
> > > > > > > > > > sure that the problems you are experiencing are not garbage  
> > > > > > > > > > collection  
> > > > > > > > > > related.
> > > > > > > > > > 
> > > > > > > > > > There are some tools that can help you diagnosing what is  
> > > > > > > > > > happening  
> > > > > > > > > > with your nodes - look at the front ends mentioned on  
> > > > > > > > > > Elasticsearch  
> > > > > > > > > > clients page (  
> > > > > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/appendix/clients.html)).  
> > > > > > > > > > Of course you can use the ones that are available on your  
> > > > > > > > > > operating  
> > > > > > > > > > system like vmstat or dstat.
> > > > > > > > > > 
> > > > > > > > > > --  
> > > > > > > > > > Regards,  
> > > > > > > > > > Rafał Kuć  
> > > > > > > > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch -  
> > > > > > > > > > Elasticsearch
> > > > > > > > > > 
> > > > > > > > > > But my question is still open, how to investigate such problems ?  
> > > > > > > > > > 🙂
> > > > > > > > > > 
> > > > > > > > > > Best regards.
> > > > > > > > > > 
> > > > > > > > > > 2012/8/15 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > > > > > > > > > No, I will check this, thanks.
> > > > > > > > > > 
> > > > > > > > > > 2012/8/15 David Pilato [david@pilato.fr](mailto:david@pilato.fr)  
> > > > > > > > > > Have you looked at the Leap second problem?
> > > > > > > > > > 
> > > > > > > > > > [http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html](http://elasticsearch-users.115913.n3.nabble.com/leapsecond-and-elasticsearch-td4019953.html)
> > > > > > > > > > 
> > > > > > > > > > ## David
> > > > > > > > > > 
> > > > > > > > > > Le 15 août 2012 à 17:10, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > > > > > > > > > 
> > > > > > > > > > > a écrit :
> > > > > > > > > > 
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > I have the following problem. I have 2 nodes. I started importing  
> > > > > > > > > > data into one of them (node A). After few hours of importing process the  
> > > > > > > > > > node B started using 100% processor and hardly answered other search and  
> > > > > > > > > > insert requests. It imported 11GB of data. How can I investigate such  
> > > > > > > > > > problems? Are there any logs that can help with such situation? What should  
> > > > > > > > > > I do ?
> > > > > > > > > > 
> > > > > > > > > > Thank you for help.
> > > > > > > > > > 
> > > > > > > > > > ## Best regards. Marcin Dojwa.
> > > > > > > > > > 
> > > > > > > > > > --
> > > > > > > > > > 
> > > > > > > > > > --
> > > > > > > > > > 
> > > > > > > > > > --
> > > > > > > > > > 
> > > > > > > > > > --
> > > > > > > > > > 
> > > > > > > > > > --
> > > > 
> > > > --
> > > > 
> > > > --
> > 
> > --
> > 
> > --
> 
> --

--  
Met vriendelijke groet,

Martijn van Groningen

--

---

<div class="post-metadata">

**Author:** ![Marcin\_Dojwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_dojwa/32/2307_2.png) [@Marcin\_Dojwa](https://discuss.elastic.co/u/Marcin_Dojwa)\
**Post date:** [October 3, 2012, 8:28pm UTC](https://discuss.elastic.co/t/100-processor-usage/8749/20 "2012-10-03T20:28:58Z")

</div>

More, I have 1 index and about 50000 aliases to this index.

W dniu środa, 3 października 2012 użytkownik Martijn v Groningen napisał:

> Hi Marcin,
> 
> Are you using one or more aliases when executing the delete by query?
> 
> Martijn
> 
> On 3 October 2012 10:50, Marcin Dojwa \<[m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)\<javascript:\_e({}, 'cvml', 'm.dojwa@livechatinc.com');\>
> 
> > wrote:
> 
> > Hi,
> > 
> > The exact mapping is here: [ES 100% processor usage · GitHub](https://gist.github.com/bbde44039b20a57eebf0)  
> > I guess that this is about chat document type because this is the only  
> > one with nested mapping. I can not give you exact data that could possibly  
> > cause such a situation because this is our clients' private data. Are you  
> > able to try reproduce the problem with this information about mapping only?  
> > If not I will try to prepare fake example data.
> > 
> > Best regards.
> > 
> > 2012/10/2 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> > 
> > It seems like its doing the delete by query with nested documents. Can  
> > you share your mapping, and sample data that you index, and the  
> > delete\_by\_query that you are using?
> > 
> > On Oct 2, 2012, at 4:16 AM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com) wrote:
> > 
> > Hi,
> > 
> > I attached 4 different (I think) snapshots of curl  
> > localhost:9200/\_nodes/hot\_threads here:  
> > [ES 100% processor usage · GitHub](https://gist.github.com/f2826104eb5b288d3fb0)
> > 
> > Best regards.  
> > Marcin Dojwa
> > 
> > 2012/9/29 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > Sure, I will try to reproduce this case on Monday. I updated ES to 0.19.9  
> > so I will check if this happens with this version too.
> > 
> > Best regards.
> > 
> > 2012/9/28 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> > 
> > Can you issue: curl localhost:9200/\_nodes/hot\_threads and gist the  
> > response?
> > 
> > On Sep 27, 2012, at 1:19 PM, Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)  
> > wrote:
> > 
> > One one thing, I am not sure if it hangs on bulk indexing or on 'delete  
> > by query' after bulk indexing.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > Fix for 1.: each bulk has 10000 inserts
> > 
> > Best regards.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > Oh, and there is nothing in gc log.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > I am using ES 0.19.8, I want to upgrade to 0.19.9 now but I am not sure  
> > if that will help.
> > 
> > 2012/9/27 Marcin Dojwa [m.dojwa@livechatinc.com](mailto:m.dojwa@livechatinc.com)
> > 
> > Hi,
> > 
> > I still need help in this 🙂 I have the following information:
> > 
> > 1. 2 nodes (es1 and es2)
> > 2. 1 index Users with 30 shards and 1 replica
> > 
> > I do the following:
> > 
> > 1. On es2 I run consecutively many bulk index operations (each bulk has  
> > 1000 inserts) and I do about 1000 such bulk inserts.
> > 2. At some point of inserting es1 uses about 300% of processor and bulk
> > 
> > --
> 
> --  
> Met vriendelijke groet,
> 
> Martijn van Groningen
> 
> --

--

[Next page](https://discuss.elastic.co/t/100-processor-usage/8749.md?page=2)