# Node works slow about 10 seconds after initialization

**URL:** <https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570>\
**Category:** Elasticsearch\
**Created:** [July 10, 2014, 9:07am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570 "2014-07-10T09:07:31Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mike\_Theairkit](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@Mike\_Theairkit](https://discuss.elastic.co/u/Mike_Theairkit)\
**Post date:** [July 10, 2014, 9:07am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/1 "2014-07-10T09:07:31Z")

</div>

Hi

I testing Elastcisearch in very simple setup:

2 nodes: 1 master, 1 slave (placed on different physical servers)  
1 index with no sharding  
Project (PHP code) access to elastcisearch cluster via haproxy.

Immediately after initialization node, when node starts to process requests,  
there are short period (about 10 seconds), when node process queries very  
slow: \>1s  
normal query time in my tests: \<100ms

In this period there are CPU usage peak (java uses up to 600% vs 30% in  
normal work)  
In this period there are no disk overload (Disk subsystem on server is  
enough fast, RAID1: 2 x ssd)

After this period, node process queries fast (\<100ms).

In case of using fixed thread pools - behavior has not changed.

State of queries in this period (search2 - slave):  
host ip bulk.active bulk.queue bulk.rejected index.active  
index.queue index.rejected search.active search.queue search.rejected  
search1 172.16.76.21 0 0 0 0  
0 0 0 0 0  
search2 172.16.76.22 0 0 0 0  
0 0 48 75 0

Slow queries:

[2014-07-10 12:59:18,456][TRACE][index.search.slowlog.fetch] [Kafka]  
[v\_v2][0] took[1s], took\_millis[1014], types[default-type], stats[],  
search\_type[QUERY\_AND\_FETCH], total\_shards[1] ...  
[2014-07-10 12:59:18,784][TRACE][index.search.slowlog.fetch] [Kafka]  
[v\_v2][0] took[1.3s], took\_millis[1313], types[default-type], stats[],  
search\_type[QUERY\_AND\_FETCH], total\_shards[1] ...

My questions:

- Is this well-known behavior of elasticsearch?
- If not - what settings can affect and eliminate this behavior?

Thanks in advance!

P.S.: cluster config in attachment.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/199c6774-87eb-4eca-acc4-6229fc613949%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/199c6774-87eb-4eca-acc4-6229fc613949%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 10, 2014, 9:18am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/2 "2014-07-10T09:18:06Z")

</div>

Unsure but I think you should set HEAP SIZE. Sounds like you are using defaults.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 10 juil. 2014 à 11:07, Mike Theairkit [theairkit@gmail.com](mailto:theairkit@gmail.com) a écrit :
> 
> Hi
> 
> I testing Elastcisearch in very simple setup:
> 
> 2 nodes: 1 master, 1 slave (placed on different physical servers)  
> 1 index with no sharding  
> Project (PHP code) access to elastcisearch cluster via haproxy.
> 
> Immediately after initialization node, when node starts to process requests,  
> there are short period (about 10 seconds), when node process queries very slow: \>1s  
> normal query time in my tests: \<100ms
> 
> In this period there are CPU usage peak (java uses up to 600% vs 30% in normal work)  
> In this period there are no disk overload (Disk subsystem on server is enough fast, RAID1: 2 x ssd)
> 
> After this period, node process queries fast (\<100ms).
> 
> In case of using fixed thread pools - behavior has not changed.
> 
> State of queries in this period (search2 - slave):  
> host ip bulk.active bulk.queue bulk.rejected index.active index.queue index.rejected search.active search.queue search.rejected  
> search1 172.16.76.21 0 0 0 0 0 0 0 0 0  
> search2 172.16.76.22 0 0 0 0 0 0 48 75 0
> 
> Slow queries:
> 
> [2014-07-10 12:59:18,456][TRACE][index.search.slowlog.fetch] [Kafka] [v\_v2][0] took[1s], took\_millis[1014], types[default-type], stats, search\_type[QUERY\_AND\_FETCH], total\_shards[1] ...  
> [2014-07-10 12:59:18,784][TRACE][index.search.slowlog.fetch] [Kafka] [v\_v2][0] took[1.3s], took\_millis[1313], types[default-type], stats, search\_type[QUERY\_AND\_FETCH], total\_shards[1] ...
> 
> My questions:
> 
> - Is this well-known behavior of elasticsearch?
> - If not - what settings can affect and eliminate this behavior?
> 
> Thanks in advance!
> 
> ## P.S.: cluster config in attachment.
> 
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/199c6774-87eb-4eca-acc4-6229fc613949%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/199c6774-87eb-4eca-acc4-6229fc613949%40googlegroups.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> \<cluster\_settings.txt\>  
> \<elasticsearch.yml\>  
> \<etc\_default\_elasticsearch.txt\>

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/AC7995A6-285F-463B-8EDD-B3691834B4F3%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/AC7995A6-285F-463B-8EDD-B3691834B4F3%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Mike\_Theairkit](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@Mike\_Theairkit](https://discuss.elastic.co/u/Mike_Theairkit)\
**Post date:** [July 10, 2014, 9:53am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/3 "2014-07-10T09:53:46Z")

</div>

Did you mean ES\_HEAP\_SIZE?

It set to 4GB (and index size, using in my tests is about 2GB).

# cat /etc/default/elasticsearch | grep HEAP

ES\_HEAP\_SIZE=4g

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/db484658-0049-4406-865d-1a5697940771%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/db484658-0049-4406-865d-1a5697940771%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 10, 2014, 2:20pm UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/4 "2014-07-10T14:20:06Z")

</div>

One of your node seems to have around 4gb allocated and the other one (the first in your node info trace) only 256m-1gb HEAP.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 10 juil. 2014 à 11:53, Mike Theairkit [theairkit@gmail.com](mailto:theairkit@gmail.com) a écrit :
> 
> Did you mean ES\_HEAP\_SIZE?
> 
> It set to 4GB (and index size, using in my tests is about 2GB).
> 
> # cat /etc/default/elasticsearch | grep HEAP
> 
> ES\_HEAP\_SIZE=4g
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/db484658-0049-4406-865d-1a5697940771%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/db484658-0049-4406-865d-1a5697940771%40googlegroups.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/C9095C1C-671B-44B9-AEB5-21B5188C24B0%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/C9095C1C-671B-44B9-AEB5-21B5188C24B0%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Mike\_Theairkit](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@Mike\_Theairkit](https://discuss.elastic.co/u/Mike_Theairkit)\
**Post date:** [July 11, 2014, 6:39am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/5 "2014-07-11T06:39:19Z")

</div>

I check my configs, now are identical, HEAP set to 4GB

Problem with slow work after initialization still persists.

Attach new cluster\_setting to this message.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/467da7e4-e9f5-46a4-a0a8-e9bb1a530c7b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/467da7e4-e9f5-46a4-a0a8-e9bb1a530c7b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 11, 2014, 7:44am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/6 "2014-07-11T07:44:01Z")

</div>

Any clue when looking at hot\_threads?

May be it's just because your segments are not warmed yet and running the first queries take some time.  
Do you have warmers?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 11 juillet 2014 à 08:39:24, Mike Theairkit ([theairkit@gmail.com](mailto:theairkit@gmail.com)) a écrit:

I check my configs, now are identical, HEAP set to 4GB

Problem with slow work after initialization still persists.

## Attach new cluster\_setting to this message.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/467da7e4-e9f5-46a4-a0a8-e9bb1a530c7b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/467da7e4-e9f5-46a4-a0a8-e9bb1a530c7b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.53bf95c1.643c9869.681%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.53bf95c1.643c9869.681%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Mike\_Theairkit](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@Mike\_Theairkit](https://discuss.elastic.co/u/Mike_Theairkit)\
**Post date:** [July 11, 2014, 8:29am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/7 "2014-07-11T08:29:06Z")

</div>

I ran tests again, and get hot\_threads when problem occurs  
See attachment.  
So, when node is slow, there are threads which using more than 60%cpu vs.  
in normal work they using ~2-5%cpu  
May you help me to interpret this log?

About warmers - I see slow work, when using warmers  
For now - there are no wamers.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f948cd4a-33ed-442b-8bf4-e4b30a61d3a6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f948cd4a-33ed-442b-8bf4-e4b30a61d3a6%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 11, 2014, 8:36am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/8 "2014-07-11T08:36:53Z")

</div>

What does your query look like?

Note: please don't add attachment to your emails but prefer put them on [gist.github.com](http://gist.github.com) for example

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 11 juillet 2014 à 10:29:11, Mike Theairkit ([theairkit@gmail.com](mailto:theairkit@gmail.com)) a écrit:

I ran tests again, and get hot\_threads when problem occurs  
See attachment.  
So, when node is slow, there are threads which using more than 60%cpu vs. in normal work they using ~2-5%cpu  
May you help me to interpret this log?

About warmers - I see slow work, when using warmers  
For now - there are no wamers.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f948cd4a-33ed-442b-8bf4-e4b30a61d3a6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f948cd4a-33ed-442b-8bf4-e4b30a61d3a6%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.53bfa225.3d1b58ba.70e%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.53bfa225.3d1b58ba.70e%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Mike\_Theairkit](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@Mike\_Theairkit](https://discuss.elastic.co/u/Mike_Theairkit)\
**Post date:** [July 11, 2014, 8:50am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/9 "2014-07-11T08:50:39Z")

</div>

Typical query: [https://gist.github.com/anonymous/20fc650ca2ada3928b0b](https://gist.github.com/anonymous/20fc650ca2ada3928b0b)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/81aaa013-e4a7-4377-9b34-bdc158c49835%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/81aaa013-e4a7-4377-9b34-bdc158c49835%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 11, 2014, 9:41am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/10 "2014-07-11T09:41:42Z")

</div>

Some comments:

Unrelated to your "issue": I would not boost by 10000. It's too much in my opinion and potentially could have opposite effect as you are expecting IIRC.  
Boosting by 1.5, 2 and 3 should be enough.  
I was expecting more OR queries but it looks like you have a reasonable list of terms here.

May be you could try some options from here:

- If only the first queries are slow, try to use warmers so the first query a user will send will get a better response time: [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html#indices-warmers](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html#indices-warmers)
- Remove highlighting and see how it performs
- Remove aggs and see how it performs

Once you have the culprit, you can try to focus on that.

HTH

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 11 juillet 2014 à 10:50:44, Mike Theairkit ([theairkit@gmail.com](mailto:theairkit@gmail.com)) a écrit:

## Typical query: [https://gist.github.com/anonymous/20fc650ca2ada3928b0b](https://gist.github.com/anonymous/20fc650ca2ada3928b0b)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/81aaa013-e4a7-4377-9b34-bdc158c49835%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/81aaa013-e4a7-4377-9b34-bdc158c49835%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.53bfb156.4db127f8.70e%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.53bfb156.4db127f8.70e%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Mike\_Theairkit](https://avatars.discourse-cdn.com/v4/letter/m/9de053/32.png) [@Mike\_Theairkit](https://discuss.elastic.co/u/Mike_Theairkit)\
**Post date:** [July 11, 2014, 10:21am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/11 "2014-07-11T10:21:59Z")

</div>

Thanks for answer!  
I will test with this options.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/90d6b137-8d88-4e36-a876-c01203b7baed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/90d6b137-8d88-4e36-a876-c01203b7baed%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:16am UTC](https://discuss.elastic.co/t/node-works-slow-about-10-seconds-after-initialization/18570/12 "2017-07-06T01:16:33Z")

</div>