Kibana Monitoring Dashboard showing wrong info

The status of every index is being wrongly shown as Deleted / Closed. But all the indices are very much in Green state.

Hi @akshaymaniyar
the status of your cluster is green if everything is correctly allocated. The state of each index doesn't directly influence the state of your cluster. If the closed index was in a yellow state your cluster will be in a yellow state.
In your case seems that your cluster state is good and the table reports that some of your indexes were deleted or closed.

It is showing this for all indices, whereas none of them are in either deleted or closed state.

Hi @akshaymaniyar

Could we please see the results of a couple queries?

On your production cluster:

GET /_cat/indices

On your monitoring cluster:

POST .monitoring-es*/_search?filter_path=aggregations.by_cluster_uuid.buckets.key,aggregations.by_cluster_uuid.buckets.by_doc_type.buckets.key,aggregations.by_cluster_uuid.buckets.by_doc_type.buckets.latest_timestamp.value_as_string
{
  "size": 0,
  "aggs": {
    "by_cluster_uuid": {
      "terms": {
        "field": "cluster_uuid",
        "size": 10
      },
      "aggs": {
        "by_doc_type": {
          "terms": {
            "field": "type",
            "size": 10
          },
          "aggs": {
            "latest_timestamp": {
              "max": {
                "field": "timestamp"
              }
            }
          }
        }
      }
    }
  }
}

Thanks!

/_cat/indices => Output

Query Output

@Mike_Place: Plz suggest what is to be done to correct the issue?

bump

@akshaymaniyar You have some many indices defined here that I wonder if perhaps you are hitting the max_bucket_size when we try to aggregate the results for the query for your index data. When you load this page, do you observe any errors in either the browser, the Kibana server log, or the Elasticsearch log where monitoring data is stored?

Yes that was indeed the issue. It is resolved by increasing the "search.max_buckets".

We were getting the below error:

Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [20000] but was [20001]. This limit can be set by changing the [search.max_buckets] cluster level setting.
	at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:110) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.consumeBucketsAndMaybeBreak(BucketsAggregator.java:132) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.terms.LongTermsAggregator.buildAggregation(LongTermsAggregator.java:148) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.AggregatorFactory$MultiBucketAggregatorWrapper.buildAggregation(AggregatorFactory.java:152) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregation(BestBucketsDeferringCollector.java:214) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.bucketAggregations(BucketsAggregator.java:141) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregation(GlobalOrdinalsStringTermsAggregator.java:238) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.AggregatorFactory$MultiBucketAggregatorWrapper.buildAggregation(AggregatorFactory.java:152) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregation(BestBucketsDeferringCollector.java:214) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.bucketAggregations(BucketsAggregator.java:141) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregation(GlobalOrdinalsStringTermsAggregator.java:238) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:130) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:121) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$18(IndicesService.java:1272) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$19(IndicesService.java:1329) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:123) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1335) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1269) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:347) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:393) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService.access$100(SearchService.java:124) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:358) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:354) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1069) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]