# Stability issues with elasticsearch cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2015, 7:12pm UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977 "2015-07-06T19:12:50Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [July 6, 2015, 7:12pm UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977/1 "2015-07-06T19:12:50Z")
Hi, Off late we've observed some stability issues with our elasticsearch cluster in production. Appreciate any kind of hints in troubleshooting this one. **Our cluster** 6 instances of c3.xlarge instances running on aws running elasticsearch 1.3.0 There is continuous indexing of small documents (~1-3kb) at the rate of ~500 per second. Documents are being written to ES from a storm cluster of 3 nodes. Aggregation queries are occasionally being made over this data spanning over a range of 30-60 mins. **Observations** The cluster state very frequently goes red and reports fewer active nodes. The state reported by each node in the cluster is different. The following log message are seen on some of the nodes: [2015-07-03 12:24:53,724][DEBUG][action.bulk] [metrics-datastore-3-QA2906-perf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-07-03 12:25:10,315][DEBUG][action.bulk] [metrics-datastore-3-QA2906-perf] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] Sometimes, just before these message we find logs like: [2015-06-24 06:39:27,679][DEBUG][action.admin.cluster.health] [metrics-datastore-1-production\_rel\_1\_7azure] connection exception while trying to forward request to master node [[metrics-datastore-3-production\_rel\_1\_7azure][4Ri9yww7RNa-TPAfLlgoNQ][ip-172-31-32-159.us-west-2.compute.internal][inet[/172.31.32.159:9300]]{max\_local\_storage\_nodes=1}], scheduling a retry. Error: [org.elasticsearch.transport.NodeDisconnectedException: [metrics-datastore-3-production\_rel\_1\_7azure][inet[/172.31.32.159:9300]][cluster/health] disconnected] [2015-06-24 06:39:28,790][INFO][discovery.zen] [metrics-datastore-1-production\_rel\_1\_7azure] master\_left [[metrics-datastore-6-production\_rel\_1\_7azure][1VJKk1OXTPW7xx5iNSkmEg][ip-172-31-19-48.us-west-2.compute.internal][inet[/172.31.19.48:9300]]{max\_local\_storage\_nodes=1}], reason [no longer master] [2015-06-24 06:39:28,790][INFO][cluster.service] [metrics-datastore-1-production\_rel\_1\_7azure] master {new [metrics-datastore-2-production\_rel\_1\_7azure][E9RKBA8yT2Wf-sp93XrTCQ][ip-172-31-0-179.us-west-2.compute.internal][inet[/172.31.0.179:9300]]{max\_local\_storage\_nodes=1}, previous [metrics-datastore-6-production\_rel\_1\_7azure][1VJKk1OXTPW7xx5iNSkmEg][ip-172-31-19-48.us-west-2.compute.internal][inet[/172.31.19.48:9300]]{max\_local\_storage\_nodes=1}}, removed {[metrics-datastore-6-production\_rel\_1\_7azure][1VJKk1OXTPW7xx5iNSkmEg][ip-172-31-19-48.us-west-2.compute.internal][inet[/172.31.19.48:9300]]{max\_local\_storage\_nodes=1},}, reason: zen-disco-master\_failed ([metrics-datastore-6-production\_rel\_1\_7azure][1VJKk1OXTPW7xx5iNSkmEg][ip-172-31-19-48.us-west-2.compute.internal][inet[/172.31.19.48:9300]]{max\_local\_storage\_nodes=1}) The node that shows these log messages remains in this state (probably for ever) until a restart. After which, the cluster state goes back to green with a consistent health reporting from all the nodes. Thanks, Srinath. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2015, 9:04pm UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977/2 "2015-07-06T21:04:51Z")
How much data in the cluster? How many shards and indices (and replicas)? You should really upgrade as well, 1.3.0 is pretty old by today's standards. ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [July 7, 2015, 12:47am UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977/3 "2015-07-07T00:47:08Z")
Thank Mark for the reply. We have already begun the upgrade activity but want to be sure that there are no other short comings leading to this situation. There are around 23 indices but only around 6 of them have considerable amount of data getting indexed. Each index is configured to have 3 shards with number of replicas (async) set to 1. The active indices (around 5 of them) have a primary storage of around 20gb but the rest have data in the order of a few megabytes. Let me know if you need any more data. ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [July 8, 2015, 3:35am UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977/4 "2015-07-08T03:35:15Z")
Bump! Anyone?.... ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 8, 2015, 4:09am UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977/5 "2015-07-08T04:09:07Z")
It could be a networking issue? ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [July 10, 2015, 4:02am UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977/6 "2015-07-10T04:02:18Z")
Don't think its a networking issue. This happens most of the time during our stability tests. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:02am UTC](https://discuss.elastic.co/t/stability-issues-with-elasticsearch-cluster/24977/7 "2017-07-06T00:02:25Z")