# Cluster unstable and running slowly **URL:** \ **Category:** Elasticsearch\ **Created:** [January 2, 2017, 9:56pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377 "2017-01-02T21:56:15Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![mvleandro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvleandro/32/14268_2.png) [@mvleandro](https://discuss.elastic.co/u/mvleandro)\ **Post date:** [January 2, 2017, 9:56pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/1 "2017-01-02T21:56:15Z")
Hello everyone! I am running a cluster with 7 nodes under version 5.1.1. I've upgraded this cluster from old 2.x version 20 days before and he was running ok until today. After today 2pm all the requests made to the any node of the cluster are delaying a lot. I checked if i have a problem with I/O and Network throughput, but it's not the case. I disabled some monitoring agents to isolate the problem. I collected some logs from my current master node and uploaded to my google drive. My cluster log: [https://drive.google.com/open?id=0B2uG9PA8RQJGX0xqNmJCUEM1Snc](https://drive.google.com/open?id=0B2uG9PA8RQJGX0xqNmJCUEM1Snc) My cluster deprecation log: [https://drive.google.com/open?id=0B2uG9PA8RQJGcEh0R1JDTG9vaU0](https://drive.google.com/open?id=0B2uG9PA8RQJGcEh0R1JDTG9vaU0) I'm not sure if the cluster upgrade caused the problem or some another factor that i can't see right now. May someone give me a light about what could be happening? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 3, 2017, 12:10pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/2 "2017-01-03T12:10:30Z")
I'm seeing a lot of date/time parsing errors in the first of those log files.. ---
**Author:** ![mvleandro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvleandro/32/14268_2.png) [@mvleandro](https://discuss.elastic.co/u/mvleandro)\ **Post date:** [January 3, 2017, 1:30pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/3 "2017-01-03T13:30:31Z")
Updating: I stopped to send data to the cluster and restarted all nodes. So i wait the cluster get green and restarted to send data to the cluster again. After this action my cluster back to running ok and i have no idea what could happened. Maybe some 5.x version bug? ---
**Author:** ![mvleandro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvleandro/32/14268_2.png) [@mvleandro](https://discuss.elastic.co/u/mvleandro)\ **Post date:** [January 3, 2017, 1:35pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/4 "2017-01-03T13:35:18Z")
Hello Mark! Thanks for your answer. So, the DateTime parsing failures are a known failures that'll be fixed soon. But i have a question: Can this failure make a total cluster stop? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 3, 2017, 1:42pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/5 "2017-01-03T13:42:19Z")
> [@mvleandro](#): > > Can this failure make a total cluster stop? Throwing an exception is one of the more expensive things you can do in Java. If you're doing this on every doc in a firehose of log records I imagine that could get costly. ---
**Author:** ![mvleandro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvleandro/32/14268_2.png) [@mvleandro](https://discuss.elastic.co/u/mvleandro)\ **Post date:** [January 3, 2017, 1:45pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/6 "2017-01-03T13:45:02Z")
Thanks again. I'll enforce this thing with teams that sending this failed requests to the cluster. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 3, 2017, 1:47pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/7 "2017-01-03T13:47:39Z")
> [@mvleandro](#): > > I'll enforce this thing with teams that sending this with extreme prejudice... ---
**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\ **Post date:** [January 3, 2017, 3:22pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/8 "2017-01-03T15:22:55Z")
Also, it seems that you are sending these requests to a master node. You should _not_ send requests to master nodes if you want better cluster stability (it is even worse if the requests generates parsing exceptions). Consider using dedicated masters (an odd number, like 1, 3, 5 - 3 dedicated masters is recommended for high availability). They don't need to be as beefy as your data nodes (2GB and 4 cores should be more than enough for your cluster) but they need to _fail independently_. Also, remember to set discovery.zen.minimum\_master\_nodes to "(total\_masters / 2) + 1". ---
**Author:** ![mvleandro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvleandro/32/14268_2.png) [@mvleandro](https://discuss.elastic.co/u/mvleandro)\ **Post date:** [January 4, 2017, 1:25pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/9 "2017-01-04T13:25:21Z")
Thank you Thiago! I'll put 3 dedicated masters on in my cluster and after that, i'll update this thread with the results. Also, i would like to know if exists a formula to calculate the ideal number of master nodes in the cluster. ---
**Author:** ![thiago](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thiago/32/32096_2.png) [@thiago](https://discuss.elastic.co/u/thiago)\ **Post date:** [January 4, 2017, 1:42pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/10 "2017-01-04T13:42:18Z")
There is no formula besides that it should be an odd number and that they _fail independently_. 3 dedicated masters will give you high availability (2 must fail at the same time for the cluster to become unavailable). ---
**Author:** ![mvleandro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvleandro/32/14268_2.png) [@mvleandro](https://discuss.elastic.co/u/mvleandro)\ **Post date:** [January 4, 2017, 1:55pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/11 "2017-01-04T13:55:40Z")
Ok. Thank you again! 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 1, 2017, 1:55pm UTC](https://discuss.elastic.co/t/cluster-unstable-and-running-slowly/70377/12 "2017-02-01T13:55:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.