Elastic search High threads , 100% utilization of non heap memory

srikanth_ramineni · May 23, 2014, 5:18am

Hi Team,

We are experiencing issue with high usage of non heap memory and high
thread count. Mostly we are seeing GC process is running. We are watching
threads using big desk from past two days. Threads are reaching peak. We
are not sure why it is reaching this much high. Two days back we ran
optimizer manually. We restarted the servers, still threads are showing
peak. Can you please suggest what might be the issues. Is there any way
to debug this kind of issue. Most of the times we are getting node not
reachable and timeout error.

[7.9gb], all_pools {[young] [3.4mb]->[64.4mb]/[266.2mb]}{[survivor]
[33.2mb]->[33.2mb]/[33.2mb]}{[old] [2.1gb]->[2.2gb]/[7.6gb]}
[2014-05-22 20:42:53,966][WARN ][monitor.jvm ] [Echo]
[gc][young][2864][37] duration [2s], collections [1]/[2.7s], total
[2s]/[23.1s], memory [2.4gb]->[2.4gb]/[7.9gb], all_pools {[young]
[104.9mb]->[3.6mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old]
[2.2gb]->[2.4gb]/[7.6gb]}
[2014-05-22 20:43:09,921][INFO ][monitor.jvm ] [Echo]
[gc][young][2865][46] duration [6.6s], collections [9]/[15.9s], total
[6.6s]/[29.7s], memory [2.4gb]->[2.6gb]/[7.9gb], all_pools {[young]
[3.6mb]->[98mb]/[266.2mb]}{[survivor] [33.2mb]->[23.9kb]/[33.2mb]}{[old]
[2.4gb]->[2.5gb]/[7.6gb]}

Below is our configuration.

Having 4 node cluster
each node having 16GB RAM -> Elastic search allocated to 10GB RAM on each
node.
Total size of data indexed : 450 GB
Each node having 350GB space

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/af014404-0b15-4387-93ca-ec5d3bdc805f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

spinscale · June 1, 2014, 8:13pm

Hey,

you should use the nodes stats and nodes info APIs to monitor elasticsearch
(or marvel, if its commercially viable for you). Also, where are you seing
the 100% utilization? Not memory but CPU wise - might make sense to add
some info here.

--Alex

On Fri, May 23, 2014 at 7:18 AM, srikanth ramineni ri.srikanth@gmail.com
wrote:

Hi Team,

We are experiencing issue with high usage of non heap memory and high
thread count. Mostly we are seeing GC process is running. We are watching
threads using big desk from past two days. Threads are reaching peak. We
are not sure why it is reaching this much high. Two days back we ran
optimizer manually. We restarted the servers, still threads are showing
peak. Can you please suggest what might be the issues. Is there any way
to debug this kind of issue. Most of the times we are getting node not
reachable and timeout error.

[7.9gb], all_pools {[young] [3.4mb]->[64.4mb]/[266.2mb]}{[survivor]
[33.2mb]->[33.2mb]/[33.2mb]}{[old] [2.1gb]->[2.2gb]/[7.6gb]}
[2014-05-22 20:42:53,966][WARN ][monitor.jvm ] [Echo]
[gc][young][2864][37] duration [2s], collections [1]/[2.7s], total
[2s]/[23.1s], memory [2.4gb]->[2.4gb]/[7.9gb], all_pools {[young]
[104.9mb]->[3.6mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old]
[2.2gb]->[2.4gb]/[7.6gb]}
[2014-05-22 20:43:09,921][INFO ][monitor.jvm ] [Echo]
[gc][young][2865][46] duration [6.6s], collections [9]/[15.9s], total
[6.6s]/[29.7s], memory [2.4gb]->[2.6gb]/[7.9gb], all_pools {[young]
[3.6mb]->[98mb]/[266.2mb]}{[survivor] [33.2mb]->[23.9kb]/[33.2mb]}{[old]
[2.4gb]->[2.5gb]/[7.6gb]}

Below is our configuration.

Having 4 node cluster
each node having 16GB RAM -> Elastic search allocated to 10GB RAM on each
node.
Total size of data indexed : 450 GB
Each node having 350GB space

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/af014404-0b15-4387-93ca-ec5d3bdc805f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/af014404-0b15-4387-93ca-ec5d3bdc805f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-0Y1ZRYbTGXYfvGh9ygeG-Q0VyoUU%3DpwQ019SoY%2BgbOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

srikanth_ramineni · June 1, 2014, 11:09pm

Hi Alex,

Thanks for your response. We are using big desk to monitor the stats.Please
find enclosed documents having screen shots for one of my node. Hope
enclosed screens shots help you to provide some inputs to us.

Thanks,
Srikanth.

On Sun, Jun 1, 2014 at 1:13 PM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

you should use the nodes stats and nodes info APIs to monitor
elasticsearch (or marvel, if its commercially viable for you). Also, where
are you seing the 100% utilization? Not memory but CPU wise - might make
sense to add some info here.

--Alex

On Fri, May 23, 2014 at 7:18 AM, srikanth ramineni ri.srikanth@gmail.com
wrote:

Hi Team,

We are experiencing issue with high usage of non heap memory and high
thread count. Mostly we are seeing GC process is running. We are watching
threads using big desk from past two days. Threads are reaching peak. We
are not sure why it is reaching this much high. Two days back we ran
optimizer manually. We restarted the servers, still threads are showing
peak. Can you please suggest what might be the issues. Is there any way
to debug this kind of issue. Most of the times we are getting node not
reachable and timeout error.

[7.9gb], all_pools {[young] [3.4mb]->[64.4mb]/[266.2mb]}{[survivor]
[33.2mb]->[33.2mb]/[33.2mb]}{[old] [2.1gb]->[2.2gb]/[7.6gb]}
[2014-05-22 20:42:53,966][WARN ][monitor.jvm ] [Echo]
[gc][young][2864][37] duration [2s], collections [1]/[2.7s], total
[2s]/[23.1s], memory [2.4gb]->[2.4gb]/[7.9gb], all_pools {[young]
[104.9mb]->[3.6mb]/[266.2mb]}{[survivor] [33.2mb]->[33.2mb]/[33.2mb]}{[old]
[2.2gb]->[2.4gb]/[7.6gb]}
[2014-05-22 20:43:09,921][INFO ][monitor.jvm ] [Echo]
[gc][young][2865][46] duration [6.6s], collections [9]/[15.9s], total
[6.6s]/[29.7s], memory [2.4gb]->[2.6gb]/[7.9gb], all_pools {[young]
[3.6mb]->[98mb]/[266.2mb]}{[survivor] [33.2mb]->[23.9kb]/[33.2mb]}{[old]
[2.4gb]->[2.5gb]/[7.6gb]}

Below is our configuration.

Having 4 node cluster
each node having 16GB RAM -> Elastic search allocated to 10GB RAM on
each node.
Total size of data indexed : 450 GB
Each node having 350GB space

Thanks,
Srikanth.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/af014404-0b15-4387-93ca-ec5d3bdc805f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/af014404-0b15-4387-93ca-ec5d3bdc805f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/EZydbbYvmXI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-0Y1ZRYbTGXYfvGh9ygeG-Q0VyoUU%3DpwQ019SoY%2BgbOg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-0Y1ZRYbTGXYfvGh9ygeG-Q0VyoUU%3DpwQ019SoY%2BgbOg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGuNUAWDAPm0by0Be7H9wHHFmeoZAsfzZ72CJZyEPYdgBExrvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.