# Timeout notification from cluster service **URL:** \ **Category:** Elasticsearch\ **Created:** [July 8, 2014, 4:23pm UTC](https://discuss.elastic.co/t/timeout-notification-from-cluster-service/18535 "2014-07-08T16:23:14Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![x0ne\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/x0ne_2/32/1405_2.png) [@x0ne\_2](https://discuss.elastic.co/u/x0ne_2)\ **Post date:** [July 8, 2014, 4:23pm UTC](https://discuss.elastic.co/t/timeout-notification-from-cluster-service/18535/1 "2014-07-08T16:23:14Z")
I am running a 4 node cluster running in EC2 and for the past few days, I have noticed that some nodes occasionally timeout on a request resulting in the following: ConnectionError(HTTPConnectionPool(host='HOST', port=9200): Read timed out. (read timeout=10)) caused by: ReadTimeoutError(HTTPConnectionPool(host='HOST', port=9200): Read timed out. (read timeout=10)) From the error, I figured it was something to do with my application keeping the elasticsearch instance open to long, so I decided to spawn it fresh every time a request was made. Still, this issue appear to pop up without much explanation. I figure it could be due to the nodes communicating, so I checked my log and saw the following: [2014-07-08 16:19:29,965][DEBUG][action.index] [Node Name] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] I see this in all of my log files, but googling around doesn't seem to lead me to a direct cause. I would guess whatever is causing the above log entry is either corresponding to the timeouts I see or is the reason they are occurring. Any ideas? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3229d028-01b1-4015-b9ae-b7e2d86faf11%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3229d028-01b1-4015-b9ae-b7e2d86faf11%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Artur\_S\_Digit](https://avatars.discourse-cdn.com/v4/letter/a/74df32/32.png) [@Artur\_S\_Digit](https://discuss.elastic.co/u/Artur_S_Digit)\ **Post date:** [August 14, 2014, 5:00am UTC](https://discuss.elastic.co/t/timeout-notification-from-cluster-service/18535/2 "2014-08-14T05:00:05Z")
I see exactly the same in my logs: [2014-08-14 04:57:43,913][DEBUG][action.index] [data-es-001] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2014-08-14 04:57:56,869][DEBUG][action.index] [data-es-001] observer timed out. notifying listener. timeout setting [1m], time since start [1m] Those lines keep appearing since we upgraded to 1.3.1. Also, Since the upgrade last week our cluster died 3 times already. I'm not sure if that's related... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1c76479d-6c28-4d73-bab2-99e9a450f79c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1c76479d-6c28-4d73-bab2-99e9a450f79c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![krishnan](https://avatars.discourse-cdn.com/v4/letter/k/a698b9/32.png) [@krishnan](https://discuss.elastic.co/u/krishnan)\ **Post date:** [August 15, 2014, 5:47am UTC](https://discuss.elastic.co/t/timeout-notification-from-cluster-service/18535/3 "2014-08-15T05:47:14Z")
Please check the status of indexes when you get these log messages. I found these log messages appear in when index status in not green. You can use following query to check : curl -s –XGET [http://10.72.12.22:9200/\_cat/indices?pretty=true](http://10.72.12.22:9200/_cat/indices?pretty=true) | grep -v green | wc -l On Thursday, August 14, 2014 10:30:05 AM UTC+5:30, Artur S. (Digit) wrote: > I see exactly the same in my logs: > [2014-08-14 04:57:43,913][DEBUG][action.index] [data-es-001] > observer: timeout notification from cluster service. timeout setting [1m], > time since start [1m] > [2014-08-14 04:57:56,869][DEBUG][action.index] [data-es-001] > observer timed out. notifying listener. timeout setting [1m], time since > start [1m] > > Those lines keep appearing since we upgraded to 1.3.1. Also, Since the > upgrade last week our cluster died 3 times already. I'm not sure if that's > related... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b4f5d04f-1674-4045-b5ba-72a2fc90db22%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b4f5d04f-1674-4045-b5ba-72a2fc90db22%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![krishnan](https://avatars.discourse-cdn.com/v4/letter/k/a698b9/32.png) [@krishnan](https://discuss.elastic.co/u/krishnan)\ **Post date:** [August 15, 2014, 5:55am UTC](https://discuss.elastic.co/t/timeout-notification-from-cluster-service/18535/4 "2014-08-15T05:55:31Z")
Please check the status of indexes when you get these log messages. I found these log messages appear in when index status in not green. You can use following query to check : curl -s –XGET http://:9200/\_cat/indices?pretty=true [http://10.72.12.22:9200/\_cat/indices?pretty=true](http://10.72.12.22:9200/_cat/indices?pretty=true) | grep -v green | wc -l On Thursday, August 14, 2014 10:30:05 AM UTC+5:30, Artur S. (Digit) wrote: > I see exactly the same in my logs: > [2014-08-14 04:57:43,913][DEBUG][action.index] [data-es-001] > observer: timeout notification from cluster service. timeout setting [1m], > time since start [1m] > [2014-08-14 04:57:56,869][DEBUG][action.index] [data-es-001] > observer timed out. notifying listener. timeout setting [1m], time since > start [1m] > > Those lines keep appearing since we upgraded to 1.3.1. Also, Since the > upgrade last week our cluster died 3 times already. I'm not sure if that's > related... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b106709-0f8b-4932-8fba-5b9d541cf77e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b106709-0f8b-4932-8fba-5b9d541cf77e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:08am UTC](https://discuss.elastic.co/t/timeout-notification-from-cluster-service/18535/5 "2017-07-06T01:08:20Z")