Node works slow about 10 seconds after initialization

Elastic Stack Elasticsearch
1

Hi

I testing Elastcisearch in very simple setup:

2 nodes: 1 master, 1 slave (placed on different physical servers)
1 index with no sharding
Project (PHP code) access to elastcisearch cluster via haproxy.

Immediately after initialization node, when node starts to process requests,
there are short period (about 10 seconds), when node process queries very
slow: >1s
normal query time in my tests: <100ms

In this period there are CPU usage peak (java uses up to 600% vs 30% in
normal work)
In this period there are no disk overload (Disk subsystem on server is
enough fast, RAID1: 2 x ssd)

After this period, node process queries fast (<100ms).

In case of using fixed thread pools - behavior has not changed.

State of queries in this period (search2 - slave):
host ip bulk.active bulk.queue bulk.rejected index.active
index.queue index.rejected search.active search.queue search.rejected
search1 172.16.76.21 0 0 0 0
0 0 0 0 0
search2 172.16.76.22 0 0 0 0
0 0 48 75 0

Slow queries:

[2014-07-10 12:59:18,456][TRACE][index.search.slowlog.fetch] [Kafka]
[v_v2][0] took[1s], took_millis[1014], types[default-type], stats[],
search_type[QUERY_AND_FETCH], total_shards[1] ...
[2014-07-10 12:59:18,784][TRACE][index.search.slowlog.fetch] [Kafka]
[v_v2][0] took[1.3s], took_millis[1313], types[default-type], stats[],
search_type[QUERY_AND_FETCH], total_shards[1] ...

My questions:

  • Is this well-known behavior of elasticsearch?
  • If not - what settings can affect and eliminate this behavior?

Thanks in advance!

P.S.: cluster config in attachment.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/199c6774-87eb-4eca-acc4-6229fc613949%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Unsure but I think you should set HEAP SIZE. Sounds like you are using defaults.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 10 juil. 2014 à 11:07, Mike Theairkit theairkit@gmail.com a écrit :

Hi

I testing Elastcisearch in very simple setup:

2 nodes: 1 master, 1 slave (placed on different physical servers)
1 index with no sharding
Project (PHP code) access to elastcisearch cluster via haproxy.

Immediately after initialization node, when node starts to process requests,
there are short period (about 10 seconds), when node process queries very slow: >1s
normal query time in my tests: <100ms

In this period there are CPU usage peak (java uses up to 600% vs 30% in normal work)
In this period there are no disk overload (Disk subsystem on server is enough fast, RAID1: 2 x ssd)

After this period, node process queries fast (<100ms).

In case of using fixed thread pools - behavior has not changed.

State of queries in this period (search2 - slave):
host ip bulk.active bulk.queue bulk.rejected index.active index.queue index.rejected search.active search.queue search.rejected
search1 172.16.76.21 0 0 0 0 0 0 0 0 0
search2 172.16.76.22 0 0 0 0 0 0 48 75 0

Slow queries:

[2014-07-10 12:59:18,456][TRACE][index.search.slowlog.fetch] [Kafka] [v_v2][0] took[1s], took_millis[1014], types[default-type], stats, search_type[QUERY_AND_FETCH], total_shards[1] ...
[2014-07-10 12:59:18,784][TRACE][index.search.slowlog.fetch] [Kafka] [v_v2][0] took[1.3s], took_millis[1313], types[default-type], stats, search_type[QUERY_AND_FETCH], total_shards[1] ...

My questions:

  • Is this well-known behavior of elasticsearch?
  • If not - what settings can affect and eliminate this behavior?

Thanks in advance!

P.S.: cluster config in attachment.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/199c6774-87eb-4eca-acc4-6229fc613949%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<cluster_settings.txt>
<elasticsearch.yml>
<etc_default_elasticsearch.txt>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/AC7995A6-285F-463B-8EDD-B3691834B4F3%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

Did you mean ES_HEAP_SIZE?

It set to 4GB (and index size, using in my tests is about 2GB).

cat /etc/default/elasticsearch | grep HEAP

ES_HEAP_SIZE=4g

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db484658-0049-4406-865d-1a5697940771%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

One of your node seems to have around 4gb allocated and the other one (the first in your node info trace) only 256m-1gb HEAP.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 10 juil. 2014 à 11:53, Mike Theairkit theairkit@gmail.com a écrit :

Did you mean ES_HEAP_SIZE?

It set to 4GB (and index size, using in my tests is about 2GB).

cat /etc/default/elasticsearch | grep HEAP

ES_HEAP_SIZE=4g

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db484658-0049-4406-865d-1a5697940771%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C9095C1C-671B-44B9-AEB5-21B5188C24B0%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

5

I check my configs, now are identical, HEAP set to 4GB

Problem with slow work after initialization still persists.

Attach new cluster_setting to this message.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/467da7e4-e9f5-46a4-a0a8-e9bb1a530c7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Any clue when looking at hot_threads?

May be it's just because your segments are not warmed yet and running the first queries take some time.
Do you have warmers?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 11 juillet 2014 à 08:39:24, Mike Theairkit (theairkit@gmail.com) a écrit:

I check my configs, now are identical, HEAP set to 4GB

Problem with slow work after initialization still persists.

Attach new cluster_setting to this message.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/467da7e4-e9f5-46a4-a0a8-e9bb1a530c7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53bf95c1.643c9869.681%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

7

I ran tests again, and get hot_threads when problem occurs
See attachment.
So, when node is slow, there are threads which using more than 60%cpu vs.
in normal work they using ~2-5%cpu
May you help me to interpret this log?

About warmers - I see slow work, when using warmers
For now - there are no wamers.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f948cd4a-33ed-442b-8bf4-e4b30a61d3a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

What does your query look like?

Note: please don't add attachment to your emails but prefer put them on gist.github.com for example

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 11 juillet 2014 à 10:29:11, Mike Theairkit (theairkit@gmail.com) a écrit:

I ran tests again, and get hot_threads when problem occurs
See attachment.
So, when node is slow, there are threads which using more than 60%cpu vs. in normal work they using ~2-5%cpu
May you help me to interpret this log?

About warmers - I see slow work, when using warmers
For now - there are no wamers.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f948cd4a-33ed-442b-8bf4-e4b30a61d3a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53bfa225.3d1b58ba.70e%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

9

Typical query: https://gist.github.com/anonymous/20fc650ca2ada3928b0b

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/81aaa013-e4a7-4377-9b34-bdc158c49835%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

Some comments:

Unrelated to your "issue": I would not boost by 10000. It's too much in my opinion and potentially could have opposite effect as you are expecting IIRC.
Boosting by 1.5, 2 and 3 should be enough.
I was expecting more OR queries but it looks like you have a reasonable list of terms here.

May be you could try some options from here:

Once you have the culprit, you can try to focus on that.

HTH

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 11 juillet 2014 à 10:50:44, Mike Theairkit (theairkit@gmail.com) a écrit:

Typical query: https://gist.github.com/anonymous/20fc650ca2ada3928b0b

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/81aaa013-e4a7-4377-9b34-bdc158c49835%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53bfb156.4db127f8.70e%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

11

Thanks for answer!
I will test with this options.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/90d6b137-8d88-4e36-a876-c01203b7baed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.