Where is my infrastructure data?


(ethr bunny) #1

Just discovered the new "infrastructure" menu item. Unfortunately, despite having many, many GB of data there doesn't seem to be anything to display here. I have plenty of dashboard data and am gathering from a slew of different *beat providers.

I've tried a variety of options in the "search for infrastructure data" field.

So - what am I doing wrong?


(Carlos Pérez Aradros) #2

Hi @ethrbunny,

Infrastructure UI relies on Metricbeat data, many of the metrics we show are available starting with
Metricbeat 6.5 so I would recommend using it. For instance, we use information from add_host_metadata to group information by host.

Best regards


(Roger) #3

so it's important to use Metricbeat 6.5? I was collecting data with Docker module on Metricbeat 6.4; Have I to update it?
here-infrastructure-UI I saw a list of points to be enabled to collect data from Docker; is it really a list or can I just enable one of them?


(ethr bunny) #4

All my hosts are on puppet and set to upgrade whenever a new version is released. They should all be on 6.5 if that's what's available.

Is there something I can check to see what's going on?


(ethr bunny) #5

FWIW: Ive checked to make sure that [metric|file]beat on my various hosts are all running 6.5.

What else should I look for?


(Carlos Pérez Aradros) #6

Hi again,

There are some things you can check to troubleshoot this issue:

  • Are you getting metricbeat data in? You should be able to see it in the Discover page, for instance
  • Are you using an index name different from the default? Infrastructure UI queries metricbeat-*

(Roger) #7

hi @exekias,
is there a way to change that index_pattern to check my index and not the default one?


(Felix Stürmer) #8

Hi @rschirin,

you can change the index pattern used to read the metrics by changing the xpack.infra.sources.default.metricAlias setting in the kibana config file. Contrary to the name, it does not have to be an alias, any index pattern supported by Elasticsearch should work.


(ethr bunny) #9

Im definitely getting metricbeat-* data


(Niklas Kurvinen) #10

Hi,

I'm seeing the same issue - infrastructure UI isn't populated ("There is no data to display.") even though I have metricbeat-* data sent from 6.5.x using the system module.

Something i noticed is that the POST requests to "graphql" return an error:

..."errors":[{"message":"[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [beat.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."...

Anyone found a solution worth sharing?


(Felix Stürmer) #11

Hi @niklaskurvinen,

that suggests a problem with the metricbeat mappings. Could you give us a sample of the output when querying

/metricbeat-*/_mapping/doc/field/beat.hostname

on your Elasticsearch API?


(Niklas Kurvinen) #12

Hi @weltenwort! Sure thing!

"metricbeat-2018.11.23" : { "mappings" : { "doc" : { "beat.hostname" : { "full_name" : "beat.hostname", "mapping" : { "hostname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } }


(Niklas Kurvinen) #13

Oh! I changed on of my nodes to not use logstash and installed the index template (using metricbeat setup) and that index is immediately picked up by the infrastructure UI.

So, now I just have to figure out how I can get logstash to use the same mapping as the index template - any tips on how to do it?


(Felix Stürmer) #14

The index template by default matches metricbeat-${VERSION}-*. I assume your logstash configuration writes to different index names? You could pick metricbeat's template from the _template/metricbeat-* API response and write it back to Elasticsearch under a different name with an index pattern that matches your logstash configuration. See the Elasticsearch docs on index templates for more usage examples of the _template API.


(Niklas Kurvinen) #15

Cheers @weltenwort, that wasn't too hard!

Everything works as expected for me now. Thanks!


(Felix Stürmer) #16

@niklaskurvinen, glad to hear it :+1:

@ethrbunny, is it possible your problems are of similar origin?


(ethr bunny) #17

Yes that's a possibility. I use metricbeat-*.

GET /metricbeat-*/_mapping/doc/field/beat.hostname

 "metricbeat-2018.11.23" : {
 "mappings" : {
  "doc" : {
    "beat.hostname" : {
      "full_name" : "beat.hostname",
      "mapping" : {
        "hostname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Is there no way to change the infrastructure lookup to use my index?


(Felix Stürmer) #18

You should be able to adapt the field to your index structure by setting

xpack.infra:
  sources:
    default:
      fields:
        hostname: 'beat.hostname.keyword'
        # pod: 'kubernetes.pod.name'
        # container: 'docker.container.name'

Depending in your indices you might also want to change the pod and container field settings I commented out above.


(ethr bunny) #19

err.. ok.. how? Where are those values set?


(Felix Stürmer) #20

Sorry, in the Kibana config file (usually called kibana.yml). The exact location of that file depends on the installation method.