Infrastructure UI is empty

Elastic Observability Metrics
1

Hello,

I'm trying to diagnose why my Infrastructure UI is empty. The waffle map is empty, but the metrics explorer is populated. I can search the metricbeat-* index without any problems. The problem arose when upgrading metricbeat to 7.x.

Kibana + Elastissearch versions: 7.3.1

Everything goes straight to Elasticsearch.

The Infrastructure UI configruration option is using "host.name" to identify hosts. And here is the mapping.

GET /metricbeat-*/_mapping/field/host.name

{
  "metricbeat-7.3.1" : {
    "mappings" : {
      "host.name" : {
        "full_name" : "host.name",
        "mapping" : {
          "name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    }
  }
}
2

Hi @luksi1,

by looking at the mapping you're already on the right track. The fact that there is a sub-field called host.name.keyword indicates that the metricbeat index template is not applied correctly.

Could you let us inspect the result of

GET /_template/metricbeat-7.3.1

?

3

Unfortunately I've already tried setting the host field to host.name.keyword. I tried again and I'm getting the same results.

I actually removed the template and removed the index to see if this was the problem, and nothing.

For the sake of completeness, I've restarted an agent to create the template and removed the metricbeat-* index and I'm still experiencing the same thing. Once again, I'm still getting nothing in the waffle map and plenty of data in the metrics view.

Here's the template (I couldn't post it here due to length):

4

Unfortunately I've already tried setting the host field to host.name.keyword.

Yes, it's not as easy as that because the metric calculations rely on other fields as well. I was just using this one field as an indicator that the mapping is incorrect in general. So from the index template you posted we can see that it matches indices with names like metricbeat-7.3.1-*. From your initial post I learned that your index is called metricbeat-7.3.1, which does not match that wildcard string. The template is therefore not applied and Elasticsearch's incorrect dynamic mapping kicks in. By default the beat indices are partitioned by day or using a increasing counter (e.g. metricbeat-7.3.1-2019.09.04 or metricbeat-7.3.1-1) to enable easy management of retention times and to increase query performance. I would recommend to stick with that, which should also lead to the correct index template being applied.

5

Well, that would make sense. But I am allowing metricbeat to create the default index, which is metricbeat-7.3.1. I'm not specifying an index in output.elasticsearch. It's simply not tagging them by default with a date or a sequential number. I can add it to our fleet of metricbeat agents via config management, but it just doesn't feel like I should have to.

6

The default setting should indeed produce compatible index names. :thinking: Would it be possible to provide the filebeat configuration (without credentials, of course)?

7

Filebeat or Metricbeat?

8

Metricbeat :innocent: sorry, I mixed it up with another thread

9

I think there's an ILM thing going on here. Here's the metricbeat.yml configuration:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.ilm.enabled: auto
setup.kibana:
  host: "https://kibana.domain.se:443"
output.elasticsearch:
  hosts: ['foo.domain.se:9200', 'bar.domain.se:9200', 'baz.domain.se:9200', 'boo.domain.se:9200', 'bah.domain.se:9200']
  protocol: "https"
  username: "elastic.beats"
  password: "my.password"
  ssl.certificate_authorities: ['/etc/metricbeat/certs/ca.pem']
  ssl.certificate: "/etc/metricbeat/certs/foo.domain.se.crt"
  ssl.key: "/etc/metricbeat/certs/foo.domain.se.key"
xpack.monitoring:
  enabled: true
  elasticsearch:
    username: "elastic.beats"
    password: "my.password"
10

If ILM was used metricbeat-7.3.1 should be an alias with the actual index being something like metricbeat-7.3.1-2019.09.04-000001. Can you check that the ILM policy referred to in the index template is installed?

GET _ilm/policy/metricbeat-7.3.1
11 
{
  "metricbeat-7.3.1" : {
    "version" : 4,
    "modified_date" : "2019-08-30T21:05:24.199Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  }
}
12

That ILM policy looks good. So my hypothesis is:

  • someone or something created an index metricbeat-7.3.1, which doesn't get the correct mapping because it doesn't match any index template
  • when starting metricbeat it can't correctly set up the metricbeat-7.3.1 rollover alias and just uses the existing index (which doesn't have the mapping)

To confirm, can you check that on startup metricbeat prints something like

INFO    [index-management]      idxmgmt/std.go:298      Write alias exists already

?

13

How did you know? :slight_smile:

Sep 4 20:46:29 vgas1607 metricbeat[17620]: 2019-09-04T20:46:29.394+0200#011INFO#011[index-management]#011idxmgmt/std.go:298#011Write alias exists already

Since you're on the money, how do I resolve this? We have a few nodes writing to that index constantly? :confused: So even if I remove this index, all the nodes hit that index immediately, not allowing me to the restart the node fast enough to create the alias and the respective indices.

14

I actually resolved it by stopping all of our agents, restarting one, and then restarting everything up. Not ideal, but it got everything going. Thanks a lot for your time.

15

Glad to hear you could resolve it.

With 7.4 the beats will actually check for that condition so startup will fail with a corresponding message: https://github.com/elastic/beats/pull/13070

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.