# Monitoring cluster bring down production **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [March 9, 2018, 2:29pm UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282 "2018-03-09T14:29:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![alepuccetti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alepuccetti/32/20174_2.png) [@alepuccetti](https://discuss.elastic.co/u/alepuccetti)\ **Post date:** [March 9, 2018, 2:29pm UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/1 "2018-03-09T14:29:17Z")
I have a separate monitoring cluster separate to the production one. However, when the monitoring cluster does not respond (e.g. maintenance, network problem, internal error) the production cluster goes RED and stop working. The kibana connected to the production get 500 errors. Is there a way to get have a less disruptive behavior by the production cluster if monitoring is unreachable? Am I doing something wrong? In each elasticsearch production node, I have this settings for xpack monitoring: ```auto xpack.monitoring.enabled: true xpack.monitoring.exporters: id1: type: http host: ["elasticsearch_domain"] auth.username: "monitoring" auth.password: "monitoring_password" ``` ---
**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\ **Post date:** [March 10, 2018, 12:47am UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/2 "2018-03-10T00:47:22Z")
Hi Alessandro, which version of Kibana are you using? I believe this has been fixed in 6.2. Thanks, CJ ---
**Author:** ![alepuccetti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alepuccetti/32/20174_2.png) [@alepuccetti](https://discuss.elastic.co/u/alepuccetti)\ **Post date:** [March 11, 2018, 7:55pm UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/3 "2018-03-11T19:55:26Z")
Hi CJ, I run 6.2 for monitoring and 5.6.4 for production. However, I do not think is a kibana problem but an Elasticsearch problem, I did not set up any xpack.monitoring variable on kibana production, I set up xpack.monitoring only on the ES nodes. ---
**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\ **Post date:** [March 12, 2018, 10:05pm UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/4 "2018-03-12T22:05:22Z")
Hi Alessandro, I'm afraid I don't understand what you mean by "6.2 for Monitoring and 5.6.4 for production." Do you mean you're using X-Pack 6.2 with Kibana 5.6.4? Thanks, CJ ---
**Author:** ![alepuccetti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alepuccetti/32/20174_2.png) [@alepuccetti](https://discuss.elastic.co/u/alepuccetti)\ **Post date:** [March 12, 2018, 11:08pm UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/5 "2018-03-12T23:08:13Z")
Hi CJ, I mean that the monitoring cluster run on a 6.2 stack (Logstash, Elasticsearch, Kibana, X-Pack) and the production cluster run on a 5.6.4 stack. ---
**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\ **Post date:** [March 13, 2018, 1:56am UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/6 "2018-03-13T01:56:57Z")
Hey Alessandro, thanks for clarifying for me. I spoke with an engineer who works on Monitoring and he thinks that you could try temporarily disabling monitoring collection in your production agent when the monitoring cluster needs some down time. You can fire a dynamic cluster setting to put the collection interval at -1 to do this: ```auto PUT /_cluster/settings { "transient" : { "xpack.monitoring.collection.interval" : "-1" } } ``` The transient setting will be reset if your cluster restarts. If you need it persistent you can just change "transient" to "persistent". He also was wondering if you could provide some logs from Elasticsearch regarding "production cluster goes RED and stops working"? That will help us figure out if this is a known issue or if it's been fixed. Thanks, CJ ---
**Author:** ![alepuccetti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alepuccetti/32/20174_2.png) [@alepuccetti](https://discuss.elastic.co/u/alepuccetti)\ **Post date:** [March 13, 2018, 8:53am UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/7 "2018-03-13T08:53:08Z")
Hi CJ, Thanks for the tip. I do not have the logs to post but I remember that there was a lot of error from the xpack.monitoring module about connection refused, which make sense because the monitoring cluster was down. The collection interval is a good workaround for schedule downtimes but for outages it will not work. It seems odd to me that a production environment stops working only because the monitoring one is not reachable. Maybe there is a good reason for that. Cheers, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 10, 2018, 8:53am UTC](https://discuss.elastic.co/t/monitoring-cluster-bring-down-production/123282/8 "2018-04-10T08:53:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.