Unexpected high CPU / IO loading

Elastic Stack Elasticsearch
1

My cluster have unexpected high loading even there is no query / indexing
recently. I suspect it is the segment merging.

  1. How can I tell if it is becuase of merging? I tried to look into the
    HotThread... but I can't really understand the meaning.
  2. I tried to turn off merging completely. That means set
    "indices.store.throttle.type" to none. It seems there is some improvement,
    but STILL swa some unexpected high loading even when idle.
  3. To compensate turning off auto segment merge, I execute
    "http://localhost:9200/_all/_optimize?max_num_segments=6" every night. Is
    this an appropriate workaround?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/634edf53-24b1-4ac1-9f35-b8ae1a3a3e91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

How high?

ES will do things int he background (mostly around lucene merging), but is
it causing a problem?

On 29 March 2015 at 13:28, Felix Ng felixng2k@gmail.com wrote:

My cluster have unexpected high loading even there is no query / indexing
recently. I suspect it is the segment merging.

  1. How can I tell if it is becuase of merging? I tried to look into the
    HotThread... but I can't really understand the meaning.
  2. I tried to turn off merging completely. That means set
    "indices.store.throttle.type" to none. It seems there is some improvement,
    but STILL swa some unexpected high loading even when idle.
  3. To compensate turning off auto segment merge, I execute "
    http://localhost:9200/_all/_optimize?max_num_segments=6" every night. Is
    this an appropriate workaround?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/634edf53-24b1-4ac1-9f35-b8ae1a3a3e91%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/634edf53-24b1-4ac1-9f35-b8ae1a3a3e91%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_rY26mFBett11Xxm92GFq4m%3DtuOTcMagV9NEMQr7ZcJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Average loading around 30 - 40.

I had 9 nodes, all Intel i7, average loading stay under 4 when ES is
'normal'.

I had attached 3 copies of HotThread results, they were captured within 20
mins.
The "Lucene Merge Thread" appears on 2 to 3 nodes, using 50-60% CPU, but as
i mentioned above, the system load-average are over 30!

On Sunday, March 29, 2015 at 3:49:05 PM UTC+8, Mark Walkom wrote:

How high?

ES will do things int he background (mostly around lucene merging), but is
it causing a problem?

On 29 March 2015 at 13:28, Felix Ng <feli...@gmail.com <javascript:>>
wrote:

My cluster have unexpected high loading even there is no query / indexing
recently. I suspect it is the segment merging.

  1. How can I tell if it is becuase of merging? I tried to look into the
    HotThread... but I can't really understand the meaning.
  2. I tried to turn off merging completely. That means set
    "indices.store.throttle.type" to none. It seems there is some improvement,
    but STILL swa some unexpected high loading even when idle.
  3. To compensate turning off auto segment merge, I execute "
    http://localhost:9200/_all/_optimize?max_num_segments=6" every night.
    Is this an appropriate workaround?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/634edf53-24b1-4ac1-9f35-b8ae1a3a3e91%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/634edf53-24b1-4ac1-9f35-b8ae1a3a3e91%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4660db9a-8733-4ae8-962c-735df932bbad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

  1. If you are not sure about merging, you should look for other reasons for
    high load. Identify the processes with high activity. Check if your storage
    I/O system can keep up.

  2. You can not turn off merging. With indices.store.throttle.type: none you
    diasble throttling.

  3. Optimizing by manual operation is always second best and not required
    every night.

Jörg

On Sun, Mar 29, 2015 at 4:28 AM, Felix Ng felixng2k@gmail.com wrote:

My cluster have unexpected high loading even there is no query / indexing
recently. I suspect it is the segment merging.

  1. How can I tell if it is becuase of merging? I tried to look into the
    HotThread... but I can't really understand the meaning.
  2. I tried to turn off merging completely. That means set
    "indices.store.throttle.type" to none. It seems there is some improvement,
    but STILL swa some unexpected high loading even when idle.
  3. To compensate turning off auto segment merge, I execute "
    http://localhost:9200/_all/_optimize?max_num_segments=6" every night. Is
    this an appropriate workaround?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/634edf53-24b1-4ac1-9f35-b8ae1a3a3e91%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/634edf53-24b1-4ac1-9f35-b8ae1a3a3e91%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEM4OTm1wUe%3Dfcy3N%3D0wgWp6hhkz%3D66tMUL%3DfMKWJzTvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.