Host metadata not parsing out correctly in ElasticSearch

Elastic Stack Kibana
1

I have Metricbeat 6.3.2 sending to Logstash 6.3.2 sending to ElasticSearch 6.3.0. I have enabled the beta add_host_metadata processor in Metricbeats.

In Kibana, the host field is showing like this.

{
  "os": {
    "family": "debian",
    "codename": "xenial",
    "version": "16.04.5 LTS (Xenial Xerus)",
    "platform": "ubuntu"
  },
  "architecture": "x86_64",
  "name": "myhost1",
  "id": "c2d8354c6253fb563beeebeb07b902b5",
  "containerized": false
}

where I would expect host.name, host.architecture, host.os.platform etc. fields to have been parsed to individual fields and be searchable.

I thought that it may have something to do with the default field mappings provided by Metricbeat which look like this.

        "host": {
          "properties": {
            "architecture": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "os": {
              "properties": {
                "family": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "platform": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "version": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            }
          }
        },

so they do not explicitly define all of the fields. However, if I reindex the documents into a new index with the same index template, I get what I expect.

I am going to edit the index template now and define all of the fields, so tomorrow when the index rolls over it will pick up the new template and I will let you know the outcome. Any thoughts in the meantime?

2

Changing the index template did not make any difference :crying_cat_face:

3

Moved to Kibana as it's clear to me now this is an issue I'm having with Kibana.

I just figured it out, it was an index pattern field mapping problem. Because I had old data which did not include the host metadata and just had the hostname as a string in the host field, Kibana was unable to resolve both mappings.

I recreated the index mappings to focus on more recent documents and the issue is fixed!

4

Glad that you posted your response back. It will be helpful for the community.

Cheers
Rashmi

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.