Kibana Monitoring Dashboard showing wrong info

akshaymaniyar · October 29, 2019, 7:23am

The status of every index is being wrongly shown as Deleted / Closed. But all the indices are very much in Green state.

markov00 · October 29, 2019, 10:40am

Hi @akshaymaniyar
the status of your cluster is green if everything is correctly allocated. The state of each index doesn't directly influence the state of your cluster. If the closed index was in a yellow state your cluster will be in a yellow state.
In your case seems that your cluster state is good and the table reports that some of your indexes were deleted or closed.

akshaymaniyar · October 29, 2019, 2:14pm

It is showing this for all indices, whereas none of them are in either deleted or closed state.

Mike_Place · October 30, 2019, 1:47pm

Hi @akshaymaniyar

Could we please see the results of a couple queries?

On your production cluster:

GET /_cat/indices

On your monitoring cluster:

POST .monitoring-es*/_search?filter_path=aggregations.by_cluster_uuid.buckets.key,aggregations.by_cluster_uuid.buckets.by_doc_type.buckets.key,aggregations.by_cluster_uuid.buckets.by_doc_type.buckets.latest_timestamp.value_as_string
{
  "size": 0,
  "aggs": {
    "by_cluster_uuid": {
      "terms": {
        "field": "cluster_uuid",
        "size": 10
      },
      "aggs": {
        "by_doc_type": {
          "terms": {
            "field": "type",
            "size": 10
          },
          "aggs": {
            "latest_timestamp": {
              "max": {
                "field": "timestamp"
              }
            }
          }
        }
      }
    }
  }
}

Thanks!

akshaymaniyar · October 31, 2019, 6:05am

/_cat/indices => Output

gist.github.com

https://gist.github.com/akshaymaniyar/e3b9682daebcc5e1d51a1ff83e00b7a4

cat indices

green open electric_deep_fryer_1                         skQJc2M1THqrXJb8fCe-Iw  1 1       231         2  482.9kb  214.8kb
green open car_dashboard_cover_1                         54zXd1V1TU-EJKh3hJs2kA  1 1      2328         6    1.1mb  580.1kb
green open fridge_lamp_cover_1                           OeOOG4v_TP-QpILAFpiXmg  1 1         5         2   87.2kb   27.3kb
green open electric_guitar_1                             yWPolTOiT-mXMPQFNujLNQ  1 1       491        13    495kb    241kb
green open car_air_ionizer_1                             WdBzq2eGQfmohIzY2upv-g  1 1       861        24    1.1mb    611kb
green open test_ndc_grocery_1                            y0k4etfaQhGUoOdPpAcF0Q  1 1       202         1  140.2kb   70.1kb
green open pet_pedometer_1                               qmAUHxvPSr-ovgy_mOfoxg  1 1         1         0   12.8kb    6.4kb
green open camera_refurbished_1                          Q7wiO8Z2T_OOrSPWZbYAPg  1 1        50        13  371.7kb  175.7kb
green open cylinder_head_dowel_pin_1                     wrf-HdcxSrWBn283fO_FEw  1 1         1         0   12.7kb    6.3kb
green open car_speaker_grill_1                           0vdr570NRBa7E82ONjDt9w  1 1         4         1     45kb   21.1kb

This file has been truncated. show original

Query Output

gist.github.com

https://gist.github.com/akshaymaniyar/ffd395f121c91099cdbf06d30270969d

Query Result

{
    "aggregations": {
        "by_cluster_uuid": {
            "buckets": [
                {
                    "key": "QYleH2E5TiulVhMJSDbDpQ",
                    "by_doc_type": {
                        "buckets": [
                            {
                                "key": "index_stats",

This file has been truncated. show original

akshaymaniyar · November 3, 2019, 1:35pm

@Mike_Place: Plz suggest what is to be done to correct the issue?

akshaymaniyar · November 5, 2019, 6:41am

bump

Mike_Place · November 13, 2019, 11:43am

@akshaymaniyar You have some many indices defined here that I wonder if perhaps you are hitting the max_bucket_size when we try to aggregate the results for the query for your index data. When you load this page, do you observe any errors in either the browser, the Kibana server log, or the Elasticsearch log where monitoring data is stored?

akshaymaniyar · November 16, 2019, 2:26pm

Yes that was indeed the issue. It is resolved by increasing the "search.max_buckets".

We were getting the below error:

Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [20000] but was [20001]. This limit can be set by changing the [search.max_buckets] cluster level setting.
	at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:110) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.consumeBucketsAndMaybeBreak(BucketsAggregator.java:132) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.terms.LongTermsAggregator.buildAggregation(LongTermsAggregator.java:148) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.AggregatorFactory$MultiBucketAggregatorWrapper.buildAggregation(AggregatorFactory.java:152) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregation(BestBucketsDeferringCollector.java:214) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.bucketAggregations(BucketsAggregator.java:141) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregation(GlobalOrdinalsStringTermsAggregator.java:238) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.AggregatorFactory$MultiBucketAggregatorWrapper.buildAggregation(AggregatorFactory.java:152) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregation(BestBucketsDeferringCollector.java:214) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.bucketAggregations(BucketsAggregator.java:141) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregation(GlobalOrdinalsStringTermsAggregator.java:238) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:130) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:121) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$18(IndicesService.java:1272) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$19(IndicesService.java:1329) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:174) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:157) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:123) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1335) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1269) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:347) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:393) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService.access$100(SearchService.java:124) ~[elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:358) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:354) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1069) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.1.jar:7.0.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]

system · December 14, 2019, 2:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.