If you see
cluster:monitor/nodes/stats[n]] request_id [82300775] timed out after
[15000ms]
in the logs, you have a monitor tool running that can not complete requests
because it takes longer than 15 seconds to traverse all the data folders on
all the nodes.
There are a number of methods to reduce disk traversal time in the data
folders:
-
switch off monitoring (not really helpful) or reduce monitor interval
(maybe helpful, maybe not)
-
increase stats request timeout (if monitor tools allow this but this does
not solve the cause of the problem)
-
monitor only an index subset of your cluster (monitor tools usually do
not have this option)
-
reduce number of segments per node -> either by optimizing indices or
adding nodes
-
wait for a fix in a future ES release
Have you counted the total number of segments? If the number is high, did
you run _optimize with max_num_segments on your indices to reduce the
number of segments?
Jörg
On Fri, Jan 9, 2015 at 6:55 AM, Revan007 dragosr@pionix.ro wrote:
Hey,
I am having trouble for some while. I am getting random node disconnects
and I cannot explain why.
There is no increase in traffic ( search or index ) when this is happening
, it feels so random to me .
I first thought it could be the aws cloud plugin so I removed it and used
unicast and pointed directly to my nodes IPs but that didn't seem to be the
problem .
I changed the type of instances, now m3.2xlarge, added more instances,
made so much modifications in ES yml config and still nothing .
Changed java oracle from 1.7 to 1.8 , changed CMS collector to G1GC and
still nothing .
I am out of ideas ... how can I get more info on what is going on ?
Here are the logs I can see from master node and the data node
http://pastebin.com/GhKfRkaa
Current config:
6 m3.x2large, 1 master, 5 data nodes.
414 indices, index/day
7372 shards. 9 shards, 1 replica per index
208 million documents, 430 GB
15 gb heap size allocated per node
ES 1.4.2
Current yml config here :
http://pastebin.com/Nmdr7F6J
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/85cc2abe-da8e-4170-8e7d-a4e01f4a22c3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHv0wxNXq_nJrj5ByxrpZmwbdiKmMUbu4YYfjuGM5XkAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.