ELK 7.3 monitoring show "No items found" for certain time range

We've build a new ES 7.3 cluster, and then notice the "Nodes" page does not correctly show data for certain time range.

Question: I want to confirm this is a known bug or this is due to our setup mistake.

  • All version is compatible (7.3)
  • Index exist. (See picture)





Hi @YOULYU_ZHANG,

It looks like your nodes might have had a lapse in reporting, based on what I'm seeing in this screenshots.

We can verify this by running the following query. You'll need to adjust the time range based on when you are and not are not seeing the data. By default, I have it running one day back (which will generate a lot of results in the response):"gte": "now-1d".

If we see lapses of time where there are no reported documents, then my best guess is something happened with those nodes and they were not reporting their monitoring data properly. Have you checked the logs for any errors?

POST .monitoring-es-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "node_stats"
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-1d"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "nodes": {
      "terms": {
        "field": "node_stats.node_id",
        "size": 10
      },
      "aggs": {
        "timestamp": {
          "terms": {
            "field": "timestamp",
            "size": 100
          }
        }
      }
    }
  }
}

Thank you for your hard work to reply.

I should also mention that the data seems correct, I've added screenshot that shows there is no missing data (at least there are data), and another screenshot that shows there is no error transmitting metric data (I've also read log, and didn't find clue).

ES and Kibana is running inside container from docker.elastic.co.

I mean If the data is missed, then the time range that we got "No items found" should shift.
but the time window of "No items found" and working correctly is always same, never shift.

I've investigated 1 to 120 minutes and find following per-minutes observation and it never change.

"No item found" time range example:

  • (13 ... 24) minutes ago to now
  • (36 ... 120) minutes ago to now

Working correctly time range example:

  • (1 ... 12) minutes ago to now
  • (25 ... 35) minutes ago to now

Notice numbers are related to 12.


If you didn't observe this phenomenon on other cluster instance, then it probably due to some error occur on our setup...:cry: I'll investigate.

By the way this is the reply of your query, I couldn't find suspicious data inside...

I got this exception when query with problematic time range:


Then I tried this Request:


and get this response:

Saying that Trying to create too many buckets.
I tried to modify xpack.monitoring.max_bucket_size from 10000 to 5000, and it does not work.
Going to investigate further.

I run

PUT /_cluster/settings
{
"persistent" : {
"search.max_buckets" : 20000
}
}

and the page works properly.

@chrisronline
Could you explain why this is happening.
I mean if I say "size": 10000 in query, then it should not create 10109 bucket, but it did...

Does that mean I have other configuration error?...
Also I suggest Kibana shouldn't act as there is no Node, it should report error to user if the monitoring query is failed. :smiley:

Can you run this and report back on the response:

POST .monitoring-es-*/_search
{
  "size": 10000,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": {
              "value": "node_stats",
              "boost": 1.0
            }
          }
        },
        {
          "term": {
            "cluster_uuid": {
              "value": "y7BPLPM4TCGLvf9Fm3z61g",
              "boost": 1.0
            }
          }
        },
        {
          "range": {
            "timestamp": {
              "from": 1565855761862,
              "to": 1565859361862,
              "include_lower": true,
              "include_upper": true,
              "format": "epoch_millis",
              "boost": 1.0
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1.0
    }
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ],
  "collapse": {
    "field": "source_node.uuid"
  }
}

BTW, there is an open issue tracking this as well: https://github.com/elastic/kibana/issues/36892

1 Like

Thanks for the link to related issue.

Here is the response whenmax_bucket is 20000

and this is the response when mab_bucket is set to null.
(Default is 10000 if I remember correctly.)

Hi @YOULYU_ZHANG,

So we discovered a nice bug, mentioned here, that I think will fix this issue up for you. I'm working on a PR today and it should be available in a release soon! Please track the issue for updates