# Monitoring interface unable to find the cluster **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-monitoring\ **Created:** [November 14, 2018, 4:16pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705 "2018-11-14T16:16:50Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 4:16pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/1 "2018-11-14T16:16:50Z")
We are currently running ELK 6.4.2 with OpenJDK 1.8.0-191 on CentOS 7.5. This cluster currently contains 30 billion docs, 9820 shards, 919 indices, 90TB of data. We were previously running 6.2.4, and one day while we were upgrading memory on the ES nodes something went sideways with the x-pack monitoring, The monitoring indices are still writing as expected but the interface complains that it cannot find the cluster. After some troubleshooting we were unable to get it working again, but we were already planning to upgrade to 6.4.x so figured we would wait till then to really dig deeper. Fast forward, we have completed upgrading all our clusters to 6.4.2, and the monitoring page on this particular cluster is still broken. The error we see is: "Monitoring Request Failed Unable to find the cluster in the selected time range. UUID: pKuY7ygvSGGF8iAR-rrVQA HTTP 404" I have looked through the data in the monitoring-\* indexes, every document has a cluster\_uuid field containing "pKuY7ygvSGGF8iAR-rrVQA". The data in the monitoring-\* indices is right up to date, I can browse the data in Kibana just fine, is seems there is some kind of mismatch between the data and what the monitoring page is expecting to find. The Error comes up almost immediately, there is no delay as if something was timing out. The monitoring indices contain 3 shards plus one replica. Increasing the time range past 1 hour still does not return any results. Any help or tips on what to look for is appreciated. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2018, 6:59pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/2 "2018-11-14T18:59:00Z")
Would you mind running the following ES query and posting the results here? ```auto POST .monitoring-es-6-*/_search { "size": 0, "aggs": { "type": { "terms": { "field": "type", "size": 20 }, "aggs": { "cluster": { "terms": { "field": "cluster_uuid", "size": 5 } } } } } } ``` ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 7:13pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/3 "2018-11-14T19:13:19Z")
Here is the output of requested query: ``` { "took": 175, "timed_out": false, "_shards": { "total": 21, "successful": 21, "skipped": 0, "failed": 0 }, "hits": { "total": 32675226, "max_score": 0, "hits": [] }, "aggregations": { "type": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "index_stats", "doc_count": 17200882, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 17200882 } ] } }, { "key": "shards", "doc_count": 15059296, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 15059296 } ] } }, { "key": "node_stats", "doc_count": 369505, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 369505 } ] } }, { "key": "index_recovery", "doc_count": 24787, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 24787 } ] } }, { "key": "indices_stats", "doc_count": 19595, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 19595 } ] } }, { "key": "cluster_stats", "doc_count": 1161, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 1161 } ] } } ] } } } ``` Thanks ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2018, 7:23pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/4 "2018-11-14T19:23:01Z")
Thanks, all of those look normal. Could you tweak that query a bit as shown below and post the results, please? ```auto POST .monitoring-es-6-*/_search { "size": 0, "aggs": { "type": { "terms": { "field": "type", "size": 20 }, "aggs": { "cluster": { "terms": { "field": "cluster_uuid", "size": 5 }, "aggs": { "latest_doc": { "top_hits": { "size": 1, "_source": "timestamp", "sort": [ { "timestamp": { "order": "desc" } } ] } } } } } } } } ``` Also, could you post a screenshot of your Kibana Monitoring Cluster Overview page as well as copy-paste the URL from the browser window while you're on that page? Thanks. ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 7:36pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/5 "2018-11-14T19:36:09Z")
Screenshot of monitoring page: ![KibanaMonitoring](https://us1.discourse-cdn.com/elastic/original/3X/8/7/87e362dbe8906256a3480827d01ff168567ab4fd.png) URL: [https://kibana.domain.com/app/monitoring#/no-data?\_g=h@44136fa](https://kibana.domain.com/app/monitoring#/no-data?_g=h@44136fa) The query output is too large for one post, will try to break it up. ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 7:38pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/6 "2018-11-14T19:38:20Z")
Query output part 1 ``` { "took": 198, "timed_out": false, "_shards": { "total": 21, "successful": 21, "skipped": 0, "failed": 0 }, "hits": { "total": 32723959, "max_score": 0, "hits": [] }, "aggregations": { "type": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "index_stats", "doc_count": 17248722, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 17248722, "latest_doc": { "hits": { "total": 17248722, "max_score": null, "hits": [ { "_index": ".monitoring-es-6-2018.11.14", "_type": "doc", "_id": "A_-zE2cBZ67Ykr9K9xAl", "_score": null, "_source": { "timestamp": "2018-11-14T19:30:18.915Z" }, "sort": [ 1542223818915 ] } ] } } } ] } }, { "key": "shards", "doc_count": 15059296, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 15059296, "latest_doc": { "hits": { "total": 15059296, "max_score": null, "hits": [ { "_index": ".monitoring-es-6-2018.11.14", "_type": "doc", "_id": "-PIaNHW1RJyPqs4TbdHOYA:DzcEcbBISHSQP36zu582Bw:logstash-2018.02.17:3:r", "_score": null, "_source": { "timestamp": "2018-11-14T19:30:19.754Z" }, "sort": [ 1542223819754 ] } ] } } } ] } }, { "key": "node_stats", "doc_count": 370294, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 370294, "latest_doc": { "hits": { "total": 370294, "max_score": null, "hits": [ { "_index": ".monitoring-es-6-2018.11.14", "_type": "doc", "_id": "9YO0E2cBJ06QRqoMB-9c", "_score": null, "_source": { "timestamp": "2018-11-14T19:30:24.525Z" }, "sort": [ 1542223824525 ] } ] } } } ] } }, ``` ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 7:38pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/7 "2018-11-14T19:38:53Z")
Part 2: ``` { "key": "index_recovery", "doc_count": 24839, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 24839, "latest_doc": { "hits": { "total": 24839, "max_score": null, "hits": [ { "_index": ".monitoring-es-6-2018.11.14", "_type": "doc", "_id": "nP-zE2cBZ67Ykr9K9xMl", "_score": null, "_source": { "timestamp": "2018-11-14T19:30:19.413Z" }, "sort": [ 1542223819413 ] } ] } } } ] } }, { "key": "indices_stats", "doc_count": 19647, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 19647, "latest_doc": { "hits": { "total": 19647, "max_score": null, "hits": [ { "_index": ".monitoring-es-6-2018.11.14", "_type": "doc", "_id": "m_-zE2cBZ67Ykr9K9xMl", "_score": null, "_source": { "timestamp": "2018-11-14T19:30:18.915Z" }, "sort": [ 1542223818915 ] } ] } } } ] } }, { "key": "cluster_stats", "doc_count": 1161, "cluster": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "pKuY7ygvSGGF8iAR-rrVQA", "doc_count": 1161, "latest_doc": { "hits": { "total": 1161, "max_score": null, "hits": [ { "_index": ".monitoring-es-6-2018.11.11", "_type": "doc", "_id": "IVGpBGcBZ67Ykr9KXP2d", "_score": null, "_source": { "timestamp": "2018-11-11T21:24:23.373Z" }, "sort": [ 1541971463373 ] } ] } } } ] } } ] } } } ``` ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [November 14, 2018, 7:39pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/8 "2018-11-14T19:39:33Z")
Hi @alaphoid, Are you using a dedicated monitoring cluster or are you using your production cluster as the monitoring cluster? More specifically, is there a config for `xpack.monitoring.elasticsearch.url` in your `kibana.yml`? ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2018, 7:40pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/9 "2018-11-14T19:40:45Z")
Thanks for the query output. I see the problem there: _most_ of the docs in the monitoring index are up-to-date. The one that isn't is of `type` = `cluster_stats`, which has the data necessary to generate the opening pages of the Monitoring UI in Kibana. Are you seeing any errors in your Elasticsearch logs, especially any that mention something about a cluster stats collector? Thanks. ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 7:43pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/10 "2018-11-14T19:43:12Z")
> [@chrisronline](#): > > Hi @alaphoid, > > Are you using a dedicated monitoring cluster or are you using your production cluster as the monitoring cluster? > > More specifically, is there a config for `xpack.monitoring.elasticsearch.url` in your `kibana.yml`? We are not using a dedicated monitoring cluster, we do not have that config line. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2018, 7:49pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/11 "2018-11-14T19:49:41Z")
> [@shaunak](#): > > Are you seeing any errors in your Elasticsearch logs, especially any that mention something about a cluster stats collector? @alaphoid, when you look in these logs, you might want to look around the timestamp of 2018-11-11T21:24:23.373Z. ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 7:52pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/13 "2018-11-14T19:52:07Z")
> [@shaunak](#): > > > [@shaunak](#): > > > > Are you seeing any errors in your Elasticsearch logs, especially any that mention something about a cluster stats collector? > > @alaphoid, when you look in these logs, you might want to look around the timestamp of 2018-11-11T21:24:23.373Z. That would have been around the time this cluster was upgraded from 6.2.4 to 6.4.2, will see if I can find anything useful. ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 8:01pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/14 "2018-11-14T20:01:02Z")
After the cluster finished allocating shards post upgrade there are no messages in the Elasticsearch logs other than garbage collection INFO messages. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2018, 8:20pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/15 "2018-11-14T20:20:35Z")
Alright, in that case I'm not really sure why the cluster stats collector stopped working around the time of upgrade. When you performed the upgrade, did the elected master node change? The cluster stats collector only runs on the elected master node so perhaps this has something to do with it. However, there are other collectors that run only on the elected master node so I'm not sure why only the cluster stats collector would be impacted. 🤔 Perhaps we could try to restart collection and see if that fixes the issue. Please try the following steps next: 1. Stop all monitoring collection by running the following query against Elasticsearch: 2. Wait about 20 seconds. Re-run the query with the long output that you ran earlier. Verify that the `timestamp`s in the output are at least 20 seconds old. This will confirm that collection has indeed stopped. 3. Start up collection again: 4. Wait about 20 seconds. Re-run the query with the long output that you ran earlier. Verify that the `timestamp`s in the output are current (or within the last 10 seconds). This will confirm that collection has indeed re-started. Especially verify that the `timestamp` nested inside the object with `"key": "cluster_stats"` is current. 5. If all timestamps are current, visit the Kibana Monitoring UI and check if that's working again. 6. If all timestamps are not current, check the Elasticsearch master node's logs for any errors and post them here. Thanks. ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 9:18pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/16 "2018-11-14T21:18:12Z")
Ok, I did the disable and enable, waited several minutes in between just to be sure, now I see some ClusterStatsCollector logs entries: ``` [2018-11-14T21:04:59,889][INFO][o.e.c.s.ClusterSettings] [fast es35] updating [xpack.monitoring.collection.enabled] from [true] to [false] [2018-11-14T21:08:07,401][INFO][o.e.c.s.ClusterSettings] [fast es35] updating [xpack.monitoring.collection.enabled] from [false] to [true] [2018-11-14T21:08:27,405][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [fast es35] collector [cluster_stats] timed out when collecting data [2018-11-14T21:08:47,406][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [fast es35] collector [cluster_stats] timed out when collecting data [2018-11-14T21:09:07,406][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [fast es35] collector [cluster_stats] timed out when collecting data ``` So it looks like a timeout is occurring, I just need to get to the bottom of that. I am seeing this timeout pretty steady every 20 seconds now. Thanks ---
**Author:** ![alaphoid](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@alaphoid](https://discuss.elastic.co/u/alaphoid)\ **Post date:** [November 14, 2018, 10:41pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/17 "2018-11-14T22:41:16Z")
So I ran this to increase the timeout: ``` PUT _cluster/settings { "persistent": { "xpack.monitoring.collection.cluster.stats.timeout": "15s" } } ``` and bingo, Its working. The timeout errors disappeared from the log and the monitoring interface is now working. So basic conclusion is cluster stats is taking longer than 10 seconds to return. I assume this is due to the index/shard count on the cluster, anything I might be able to do or look at to speed it up? Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 12, 2018, 10:41pm UTC](https://discuss.elastic.co/t/monitoring-interface-unable-to-find-the-cluster/156705/18 "2018-12-12T22:41:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.