Metrics -> Inventory tab shows all zero's, but Metrics -> Explorer shows data

Elastic Observability Metrics
1

I'm collecting metrics via metricbeat.

The inventory shows all 27 servers as boxes, but with 0 as the value for each server and each indicator. However, when I click on the Explorer tab I can create a graph showing data for each indicator.

Also, using the Discovery section, I can see metrics coming in via metricbeat-*.

Any tips on how to trouble shoot?

2

The inventory view is displaying the last bucket in the date histogram for those metrics. I wonder if there is a lag in the data that would cause the last bucket in the date histogram to be empty?

Another issue could be the time offset somewhere between the client, server, and your browser; this is really rare but I've seen it happen in the past. I would hate to not mention it and something simple like this be the culprit.

3

Oh - can you say more about the "last bucket in the data histogram" theory?

There's been an epic battle between my feeble intelligence and the index lifetime management system (and I seem to be losing). But in the process I have deleted and re-created the metricbeat indices several times (as I start to get the upper hand).

Maybe that has something to do with it? I did wait about 12 hours - and even after that length of time, the metrics show 0 accross the board. Do I need to wait longer?

4

Sorry... I meant date histogram. When you're looking at the Host CPU Usage metric in the Inventory View, we query for the last 20 minutes of data and we create 20 1 minute buckets with the date histogram aggregation. Let's say the current time is 10.30, the date histogram would cover from 10:10 to 10:30 and every bucket would represent 1 minute of data. If the data in your index stops at 10:28, the last 3 minutes (or last 3 buckets) of the date histogram will be empty.

5

Thanks for the added info. Unfortunately, I'm still stuck. It's been a full day, so I don't think we are experiencing any 20 minute delays. Our servers are all set to UTC and my laptop is set to America/New_York, but in the Discover section, time stamps seem to be properly converted to America/New_York so I don't think that's the problem.

Do you have any other suggestions for how to debug?

6

I think the problem may be that I deleted and then re-created my metricbeat-* index pattern. Could that be the reason? Is there a way to reset the metrics snapshot to use the new index pattern?

7

Problem solved - at last. I finally got the clue from the elasticsearch log:

[2020-12-05T02:59:29,729][WARN ][o.e.c.m.MetadataIndexTemplateService] [log001] legacy template [metricbeat-7.10.0] has index patterns [metricbeat-*] matching patterns from existing composable templates [metricbeat-7.9.2] with patterns (metricbeat-7.9.2 => [metricbeat-*]); this template [metricbeat-7.10.0] may be ignored in favor of a composable template at index creation time

At some point in my debugging I think I added a composable template. And that stuck around, poisoning the legacy template. Once I got rid of that errant composable template, all the data starting coming through just fine.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.