Where is my infrastructure data?

Elastic Observability Metrics
1

Just discovered the new "infrastructure" menu item. Unfortunately, despite having many, many GB of data there doesn't seem to be anything to display here. I have plenty of dashboard data and am gathering from a slew of different *beat providers.

I've tried a variety of options in the "search for infrastructure data" field.

So - what am I doing wrong?

2

Hi @ethrbunny,

Infrastructure UI relies on Metricbeat data, many of the metrics we show are available starting with
Metricbeat 6.5 so I would recommend using it. For instance, we use information from add_host_metadata to group information by host.

Best regards

3

so it's important to use Metricbeat 6.5? I was collecting data with Docker module on Metricbeat 6.4; Have I to update it?
here-infrastructure-UI I saw a list of points to be enabled to collect data from Docker; is it really a list or can I just enable one of them?

4

All my hosts are on puppet and set to upgrade whenever a new version is released. They should all be on 6.5 if that's what's available.

Is there something I can check to see what's going on?

5

FWIW: Ive checked to make sure that [metric|file]beat on my various hosts are all running 6.5.

What else should I look for?

6

Hi again,

There are some things you can check to troubleshoot this issue:

  • Are you getting metricbeat data in? You should be able to see it in the Discover page, for instance
  • Are you using an index name different from the default? Infrastructure UI queries metricbeat-*
7

hi @exekias,
is there a way to change that index_pattern to check my index and not the default one?

8

Hi @rschirin,

you can change the index pattern used to read the metrics by changing the xpack.infra.sources.default.metricAlias setting in the kibana config file. Contrary to the name, it does not have to be an alias, any index pattern supported by Elasticsearch should work.

1 Like
9

Im definitely getting metricbeat-* data

1 Like
10

Hi,

I'm seeing the same issue - infrastructure UI isn't populated ("There is no data to display.") even though I have metricbeat-* data sent from 6.5.x using the system module.

Something i noticed is that the POST requests to "graphql" return an error:

..."errors":[{"message":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [beat.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."...

Anyone found a solution worth sharing?

11

Hi @niklaskurvinen,

that suggests a problem with the metricbeat mappings. Could you give us a sample of the output when querying

/metricbeat-*/_mapping/doc/field/beat.hostname

on your Elasticsearch API?

12

Hi @weltenwort! Sure thing!

"metricbeat-2018.11.23" : { "mappings" : { "doc" : { "beat.hostname" : { "full_name" : "beat.hostname", "mapping" : { "hostname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } }

13

Oh! I changed on of my nodes to not use logstash and installed the index template (using metricbeat setup) and that index is immediately picked up by the infrastructure UI.

So, now I just have to figure out how I can get logstash to use the same mapping as the index template - any tips on how to do it?

14

The index template by default matches metricbeat-${VERSION}-*. I assume your logstash configuration writes to different index names? You could pick metricbeat's template from the _template/metricbeat-* API response and write it back to Elasticsearch under a different name with an index pattern that matches your logstash configuration. See the Elasticsearch docs on index templates for more usage examples of the _template API.

15

Cheers @weltenwort, that wasn't too hard!

Everything works as expected for me now. Thanks!

16

@niklaskurvinen, glad to hear it :+1:

@ethrbunny, is it possible your problems are of similar origin?

1 Like
17

Yes that's a possibility. I use metricbeat-*.

GET /metricbeat-*/_mapping/doc/field/beat.hostname

 "metricbeat-2018.11.23" : {
 "mappings" : {
  "doc" : {
    "beat.hostname" : {
      "full_name" : "beat.hostname",
      "mapping" : {
        "hostname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Is there no way to change the infrastructure lookup to use my index?

18

You should be able to adapt the field to your index structure by setting

xpack.infra:
  sources:
    default:
      fields:
        hostname: 'beat.hostname.keyword'
        # pod: 'kubernetes.pod.name'
        # container: 'docker.container.name'

Depending in your indices you might also want to change the pod and container field settings I commented out above.

19

err.. ok.. how? Where are those values set?

20

Sorry, in the Kibana config file (usually called kibana.yml). The exact location of that file depends on the installation method.