Monitoring interface unable to find the cluster

alaphoid · November 14, 2018, 4:16pm

We are currently running ELK 6.4.2 with OpenJDK 1.8.0-191 on CentOS 7.5. This cluster currently contains 30 billion docs, 9820 shards, 919 indices, 90TB of data.

We were previously running 6.2.4, and one day while we were upgrading memory on the ES nodes something went sideways with the x-pack monitoring, The monitoring indices are still writing as expected but the interface complains that it cannot find the cluster. After some troubleshooting we were unable to get it working again, but we were already planning to upgrade to 6.4.x so figured we would wait till then to really dig deeper. Fast forward, we have completed upgrading all our clusters to 6.4.2, and the monitoring page on this particular cluster is still broken.

The error we see is:
"Monitoring Request Failed
Unable to find the cluster in the selected time range. UUID: pKuY7ygvSGGF8iAR-rrVQA
HTTP 404"

I have looked through the data in the monitoring-* indexes, every document has a cluster_uuid field containing "pKuY7ygvSGGF8iAR-rrVQA".

The data in the monitoring-* indices is right up to date, I can browse the data in Kibana just fine, is seems there is some kind of mismatch between the data and what the monitoring page is expecting to find. The Error comes up almost immediately, there is no delay as if something was timing out. The monitoring indices contain 3 shards plus one replica. Increasing the time range past 1 hour still does not return any results.

Any help or tips on what to look for is appreciated.

shaunak · November 14, 2018, 6:59pm

Would you mind running the following ES query and posting the results here?

POST .monitoring-es-6-*/_search
{
  "size": 0,
  "aggs": {
    "type": {
      "terms": {
        "field": "type",
        "size": 20
      },
      "aggs": {
        "cluster": {
          "terms": {
            "field": "cluster_uuid",
            "size": 5
          }
        }
      }
    }
  }
}

alaphoid · November 14, 2018, 7:13pm

Here is the output of requested query:

{
  "took": 175,
  "timed_out": false,
  "_shards": {
    "total": 21,
    "successful": 21,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 32675226,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "type": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "index_stats",
          "doc_count": 17200882,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 17200882
              }
            ]
          }
        },
        {
          "key": "shards",
          "doc_count": 15059296,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 15059296
              }
            ]
          }
        },
        {
          "key": "node_stats",
          "doc_count": 369505,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 369505
              }
            ]
          }
        },
        {
          "key": "index_recovery",
          "doc_count": 24787,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 24787
              }
            ]
          }
        },
        {
          "key": "indices_stats",
          "doc_count": 19595,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 19595
              }
            ]
          }
        },
        {
          "key": "cluster_stats",
          "doc_count": 1161,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 1161
              }
            ]
          }
        }
      ]
    }
  }
}

Thanks

shaunak · November 14, 2018, 7:23pm

Thanks, all of those look normal. Could you tweak that query a bit as shown below and post the results, please?

POST .monitoring-es-6-*/_search
{
  "size": 0,
  "aggs": {
    "type": {
      "terms": {
        "field": "type",
        "size": 20
      },
      "aggs": {
        "cluster": {
          "terms": {
            "field": "cluster_uuid",
            "size": 5
          },
          "aggs": {
            "latest_doc": {
              "top_hits": {
                "size": 1,
                "_source": "timestamp",
                "sort": [
                  {
                    "timestamp": {
                      "order": "desc"
                    }
                  }
                ]
              }
            }
          }
        }
      }
    }
  }
}

Also, could you post a screenshot of your Kibana Monitoring Cluster Overview page as well as copy-paste the URL from the browser window while you're on that page?

Thanks.

alaphoid · November 14, 2018, 7:36pm

Screenshot of monitoring page:
KibanaMonitoring

URL:
https://kibana.domain.com/app/monitoring#/no-data?_g=h@44136fa

The query output is too large for one post, will try to break it up.

alaphoid · November 14, 2018, 7:38pm

Query output part 1

{
  "took": 198,
  "timed_out": false,
  "_shards": {
    "total": 21,
    "successful": 21,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 32723959,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "type": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "index_stats",
          "doc_count": 17248722,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 17248722,
                "latest_doc": {
                  "hits": {
                    "total": 17248722,
                    "max_score": null,
                    "hits": [
                      {
                        "_index": ".monitoring-es-6-2018.11.14",
                        "_type": "doc",
                        "_id": "A_-zE2cBZ67Ykr9K9xAl",
                        "_score": null,
                        "_source": {
                          "timestamp": "2018-11-14T19:30:18.915Z"
                        },
                        "sort": [
                          1542223818915
                        ]
                      }
                    ]
                  }
                }
              }
            ]
          }
        },
        {
          "key": "shards",
          "doc_count": 15059296,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 15059296,
                "latest_doc": {
                  "hits": {
                    "total": 15059296,
                    "max_score": null,
                    "hits": [
                      {
                        "_index": ".monitoring-es-6-2018.11.14",
                        "_type": "doc",
                        "_id": "-PIaNHW1RJyPqs4TbdHOYA:DzcEcbBISHSQP36zu582Bw:logstash-2018.02.17:3:r",
                        "_score": null,
                        "_source": {
                          "timestamp": "2018-11-14T19:30:19.754Z"
                        },
                        "sort": [
                          1542223819754
                        ]
                      }
                    ]
                  }
                }
              }
            ]
          }
        },
        {
          "key": "node_stats",
          "doc_count": 370294,
          "cluster": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "pKuY7ygvSGGF8iAR-rrVQA",
                "doc_count": 370294,
                "latest_doc": {
                  "hits": {
                    "total": 370294,
                    "max_score": null,
                    "hits": [
                      {
                        "_index": ".monitoring-es-6-2018.11.14",
                        "_type": "doc",
                        "_id": "9YO0E2cBJ06QRqoMB-9c",
                        "_score": null,
                        "_source": {
                          "timestamp": "2018-11-14T19:30:24.525Z"
                        },
                        "sort": [
                          1542223824525
                        ]
                      }
                    ]
                  }
                }
              }
            ]
          }
        },

alaphoid · November 14, 2018, 7:38pm

Part 2:

{
  "key": "index_recovery",
  "doc_count": 24839,
  "cluster": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 0,
    "buckets": [
      {
        "key": "pKuY7ygvSGGF8iAR-rrVQA",
        "doc_count": 24839,
        "latest_doc": {
          "hits": {
            "total": 24839,
            "max_score": null,
            "hits": [
              {
                "_index": ".monitoring-es-6-2018.11.14",
                "_type": "doc",
                "_id": "nP-zE2cBZ67Ykr9K9xMl",
                "_score": null,
                "_source": {
                  "timestamp": "2018-11-14T19:30:19.413Z"
                },
                "sort": [
                  1542223819413
                ]
              }
            ]
          }
        }
      }
    ]
  }
},
{
  "key": "indices_stats",
  "doc_count": 19647,
  "cluster": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 0,
    "buckets": [
      {
        "key": "pKuY7ygvSGGF8iAR-rrVQA",
        "doc_count": 19647,
        "latest_doc": {
          "hits": {
            "total": 19647,
            "max_score": null,
            "hits": [
              {
                "_index": ".monitoring-es-6-2018.11.14",
                "_type": "doc",
                "_id": "m_-zE2cBZ67Ykr9K9xMl",
                "_score": null,
                "_source": {
                  "timestamp": "2018-11-14T19:30:18.915Z"
                },
                "sort": [
                  1542223818915
                ]
              }
            ]
          }
        }
      }
    ]
  }
},
{
  "key": "cluster_stats",
  "doc_count": 1161,
  "cluster": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 0,
    "buckets": [
      {
        "key": "pKuY7ygvSGGF8iAR-rrVQA",
        "doc_count": 1161,
        "latest_doc": {
          "hits": {
            "total": 1161,
            "max_score": null,
            "hits": [
              {
                "_index": ".monitoring-es-6-2018.11.11",
                "_type": "doc",
                "_id": "IVGpBGcBZ67Ykr9KXP2d",
                "_score": null,
                "_source": {
                  "timestamp": "2018-11-11T21:24:23.373Z"
                },
                "sort": [
                  1541971463373
                ]
              }
            ]
          }
        }
      }
    ]
  }
}
  ]
}
  }
}

chrisronline · November 14, 2018, 7:39pm

Hi @alaphoid,

Are you using a dedicated monitoring cluster or are you using your production cluster as the monitoring cluster?

More specifically, is there a config for xpack.monitoring.elasticsearch.url in your kibana.yml?

shaunak · November 14, 2018, 7:40pm

Thanks for the query output. I see the problem there: most of the docs in the monitoring index are up-to-date. The one that isn't is of type = cluster_stats, which has the data necessary to generate the opening pages of the Monitoring UI in Kibana.

Are you seeing any errors in your Elasticsearch logs, especially any that mention something about a cluster stats collector?

Thanks.

alaphoid · November 14, 2018, 7:43pm

We are not using a dedicated monitoring cluster, we do not have that config line.

shaunak · November 14, 2018, 7:49pm

@alaphoid, when you look in these logs, you might want to look around the timestamp of 2018-11-11T21:24:23.373Z.

alaphoid · November 14, 2018, 7:52pm

That would have been around the time this cluster was upgraded from 6.2.4 to 6.4.2, will see if I can find anything useful.

alaphoid · November 14, 2018, 8:01pm

After the cluster finished allocating shards post upgrade there are no messages in the Elasticsearch logs other than garbage collection INFO messages.

shaunak · November 14, 2018, 8:20pm

Alright, in that case I'm not really sure why the cluster stats collector stopped working around the time of upgrade. When you performed the upgrade, did the elected master node change? The cluster stats collector only runs on the elected master node so perhaps this has something to do with it. However, there are other collectors that run only on the elected master node so I'm not sure why only the cluster stats collector would be impacted.

Perhaps we could try to restart collection and see if that fixes the issue. Please try the following steps next:

Stop all monitoring collection by running the following query against Elasticsearch:

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": false
  }
}

Wait about 20 seconds. Re-run the query with the long output that you ran earlier. Verify that the timestamps in the output are at least 20 seconds old. This will confirm that collection has indeed stopped.

Start up collection again:

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

Wait about 20 seconds. Re-run the query with the long output that you ran earlier. Verify that the timestamps in the output are current (or within the last 10 seconds). This will confirm that collection has indeed re-started. Especially verify that the timestamp nested inside the object with "key": "cluster_stats" is current.
If all timestamps are current, visit the Kibana Monitoring UI and check if that's working again.
If all timestamps are not current, check the Elasticsearch master node's logs for any errors and post them here.

Thanks.

alaphoid · November 14, 2018, 9:18pm

Ok, I did the disable and enable, waited several minutes in between just to be sure, now I see some ClusterStatsCollector logs entries:

[2018-11-14T21:04:59,889][INFO ][o.e.c.s.ClusterSettings  ] [fast es35] updating [xpack.monitoring.collection.enabled] from [true] to [false]
[2018-11-14T21:08:07,401][INFO ][o.e.c.s.ClusterSettings  ] [fast es35] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2018-11-14T21:08:27,405][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [fast es35] collector [cluster_stats] timed out when collecting data
[2018-11-14T21:08:47,406][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [fast es35] collector [cluster_stats] timed out when collecting data
[2018-11-14T21:09:07,406][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [fast es35] collector [cluster_stats] timed out when collecting data

So it looks like a timeout is occurring, I just need to get to the bottom of that. I am seeing this timeout pretty steady every 20 seconds now.

Thanks

alaphoid · November 14, 2018, 10:41pm

So I ran this to increase the timeout:

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.cluster.stats.timeout": "15s"
  }
}

and bingo, Its working. The timeout errors disappeared from the log and the monitoring interface is now working. So basic conclusion is cluster stats is taking longer than 10 seconds to return. I assume this is due to the index/shard count on the cluster, anything I might be able to do or look at to speed it up?

Thanks

system · December 12, 2018, 10:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Stack-monitoring Kibana elastic-stack-monitoring	5	383	April 21, 2022
Monitoring Request Failed Kibana elastic-stack-monitoring	3	2477	February 2, 2021
Unable to find the cluster in the selected time range Kibana elastic-stack-monitoring	2	1373	July 15, 2021
Unable to find the cluster in the selected time range. HTTP 404 Kibana elastic-stack-monitoring	6	7679	February 11, 2021
Stack monitoring does not show the cluster information and has some errors Kibana elastic-stack-monitoring	2	967	April 14, 2021

Monitoring interface unable to find the cluster

Related Topics