Timeout notification from cluster service

Elastic Stack Elasticsearch
1

I am running a 4 node cluster running in EC2 and for the past few days, I
have noticed that some nodes occasionally timeout on a request resulting in
the following:

ConnectionError(HTTPConnectionPool(host='HOST', port=9200): Read timed out.
(read timeout=10)) caused by:
ReadTimeoutError(HTTPConnectionPool(host='HOST', port=9200): Read timed
out. (read timeout=10))

From the error, I figured it was something to do with my application
keeping the elasticsearch instance open to long, so I decided to spawn it
fresh every time a request was made. Still, this issue appear to pop up
without much explanation. I figure it could be due to the nodes
communicating, so I checked my log and saw the following:

[2014-07-08 16:19:29,965][DEBUG][action.index ] [Node Name]
observer: timeout notification from cluster service. timeout setting [1m],
time since start [1m]

I see this in all of my log files, but googling around doesn't seem to lead
me to a direct cause. I would guess whatever is causing the above log entry
is either corresponding to the timeouts I see or is the reason they are
occurring. Any ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3229d028-01b1-4015-b9ae-b7e2d86faf11%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I see exactly the same in my logs:
[2014-08-14 04:57:43,913][DEBUG][action.index ] [data-es-001]
observer: timeout notification from cluster service. timeout setting [1m],
time since start [1m]
[2014-08-14 04:57:56,869][DEBUG][action.index ] [data-es-001]
observer timed out. notifying listener. timeout setting [1m], time since
start [1m]

Those lines keep appearing since we upgraded to 1.3.1. Also, Since the
upgrade last week our cluster died 3 times already. I'm not sure if that's
related...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c76479d-6c28-4d73-bab2-99e9a450f79c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Please check the status of indexes when you get these log messages. I found
these log messages appear in when index status in not green.
You can use following query to check : curl -s –XGET
http://10.72.12.22:9200/_cat/indices?pretty=true | grep -v green | wc -l

On Thursday, August 14, 2014 10:30:05 AM UTC+5:30, Artur S. (Digit) wrote:

I see exactly the same in my logs:
[2014-08-14 04:57:43,913][DEBUG][action.index ] [data-es-001]
observer: timeout notification from cluster service. timeout setting [1m],
time since start [1m]
[2014-08-14 04:57:56,869][DEBUG][action.index ] [data-es-001]
observer timed out. notifying listener. timeout setting [1m], time since
start [1m]

Those lines keep appearing since we upgraded to 1.3.1. Also, Since the
upgrade last week our cluster died 3 times already. I'm not sure if that's
related...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b4f5d04f-1674-4045-b5ba-72a2fc90db22%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Please check the status of indexes when you get these log messages. I found
these log messages appear in when index status in not green.
You can use following query to check : curl -s –XGET
http://:9200/_cat/indices?pretty=true
http://10.72.12.22:9200/_cat/indices?pretty=true | grep -v green | wc -l

On Thursday, August 14, 2014 10:30:05 AM UTC+5:30, Artur S. (Digit) wrote:

I see exactly the same in my logs:
[2014-08-14 04:57:43,913][DEBUG][action.index ] [data-es-001]
observer: timeout notification from cluster service. timeout setting [1m],
time since start [1m]
[2014-08-14 04:57:56,869][DEBUG][action.index ] [data-es-001]
observer timed out. notifying listener. timeout setting [1m], time since
start [1m]

Those lines keep appearing since we upgraded to 1.3.1. Also, Since the
upgrade last week our cluster died 3 times already. I'm not sure if that's
related...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4b106709-0f8b-4932-8fba-5b9d541cf77e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.