ELK 7.3 monitoring show "No items found" for certain time range

Elastic Stack Kibana
1

We've build a new ES 7.3 cluster, and then notice the "Nodes" page does not correctly show data for certain time range.

Question: I want to confirm this is a known bug or this is due to our setup mistake.

  • All version is compatible (7.3)
  • Index exist. (See picture)

41
Screen Shot 2019-08-14 at 12.25.41.png3034×1390 177 KB

30
Screen Shot 2019-08-14 at 12.26.30.png3008×854 150 KB

24
Screen Shot 2019-08-14 at 12.36.24.png2966×942 206 KB

49
Screen Shot 2019-08-14 at 12.37.49.png3016×996 218 KB

27
Screen Shot 2019-08-14 at 12.38.27.png2022×326 91.8 KB

2

Hi @YOULYU_ZHANG,

It looks like your nodes might have had a lapse in reporting, based on what I'm seeing in this screenshots.

We can verify this by running the following query. You'll need to adjust the time range based on when you are and not are not seeing the data. By default, I have it running one day back (which will generate a lot of results in the response):"gte": "now-1d".

If we see lapses of time where there are no reported documents, then my best guess is something happened with those nodes and they were not reporting their monitoring data properly. Have you checked the logs for any errors?

POST .monitoring-es-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "node_stats"
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-1d"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "nodes": {
      "terms": {
        "field": "node_stats.node_id",
        "size": 10
      },
      "aggs": {
        "timestamp": {
          "terms": {
            "field": "timestamp",
            "size": 100
          }
        }
      }
    }
  }
}
3

Thank you for your hard work to reply.

I should also mention that the data seems correct, I've added screenshot that shows there is no missing data (at least there are data), and another screenshot that shows there is no error transmitting metric data (I've also read log, and didn't find clue).

ES and Kibana is running inside container from docker.elastic.co.

I mean If the data is missed, then the time range that we got "No items found" should shift.
but the time window of "No items found" and working correctly is always same, never shift.

I've investigated 1 to 120 minutes and find following per-minutes observation and it never change.

"No item found" time range example:

  • (13 ... 24) minutes ago to now
  • (36 ... 120) minutes ago to now

Working correctly time range example:

  • (1 ... 12) minutes ago to now
  • (25 ... 35) minutes ago to now

Notice numbers are related to 12.

33
Screen Shot 2019-08-15 at 13.29.33.png5010×660 230 KB

05
Screen Shot 2019-08-15 at 13.32.05.png3838×810 321 KB

If you didn't observe this phenomenon on other cluster instance, then it probably due to some error occur on our setup...:cry: I'll investigate.

By the way this is the reply of your query, I couldn't find suspicious data inside...

4

I got this exception when query with problematic time range:


30
Screen Shot 2019-08-15 at 17.47.30.png3486×1198 752 KB

Then I tried this Request:


and get this response:

Saying that Trying to create too many buckets.
I tried to modify xpack.monitoring.max_bucket_size from 10000 to 5000, and it does not work.
Going to investigate further.

5

I run

PUT /_cluster/settings
{
"persistent" : {
"search.max_buckets" : 20000
}
}

and the page works properly.

@chrisronline
Could you explain why this is happening.
I mean if I say "size": 10000 in query, then it should not create 10109 bucket, but it did...

Does that mean I have other configuration error?...
Also I suggest Kibana shouldn't act as there is no Node, it should report error to user if the monitoring query is failed. :smiley:

6

Can you run this and report back on the response:

POST .monitoring-es-*/_search
{
  "size": 10000,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": {
              "value": "node_stats",
              "boost": 1.0
            }
          }
        },
        {
          "term": {
            "cluster_uuid": {
              "value": "y7BPLPM4TCGLvf9Fm3z61g",
              "boost": 1.0
            }
          }
        },
        {
          "range": {
            "timestamp": {
              "from": 1565855761862,
              "to": 1565859361862,
              "include_lower": true,
              "include_upper": true,
              "format": "epoch_millis",
              "boost": 1.0
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1.0
    }
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ],
  "collapse": {
    "field": "source_node.uuid"
  }
}
7

BTW, there is an open issue tracking this as well: https://github.com/elastic/kibana/issues/36892

1 Like
8

Thanks for the link to related issue.

Here is the response whenmax_bucket is 20000

and this is the response when mab_bucket is set to null.
(Default is 10000 if I remember correctly.)

9

Hi @YOULYU_ZHANG,

So we discovered a nice bug, mentioned here, that I think will fix this issue up for you. I'm working on a PR today and it should be available in a release soon! Please track the issue for updates

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.