Host metadata not parsing out correctly in ElasticSearch

I have Metricbeat 6.3.2 sending to Logstash 6.3.2 sending to ElasticSearch 6.3.0. I have enabled the beta add_host_metadata processor in Metricbeats.

In Kibana, the host field is showing like this.

{
  "os": {
    "family": "debian",
    "codename": "xenial",
    "version": "16.04.5 LTS (Xenial Xerus)",
    "platform": "ubuntu"
  },
  "architecture": "x86_64",
  "name": "myhost1",
  "id": "c2d8354c6253fb563beeebeb07b902b5",
  "containerized": false
}

where I would expect host.name, host.architecture, host.os.platform etc. fields to have been parsed to individual fields and be searchable.

I thought that it may have something to do with the default field mappings provided by Metricbeat which look like this.

        "host": {
          "properties": {
            "architecture": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "os": {
              "properties": {
                "family": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "platform": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "version": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            }
          }
        },

so they do not explicitly define all of the fields. However, if I reindex the documents into a new index with the same index template, I get what I expect.

I am going to edit the index template now and define all of the fields, so tomorrow when the index rolls over it will pick up the new template and I will let you know the outcome. Any thoughts in the meantime?

Changing the index template did not make any difference :crying_cat_face:

Moved to Kibana as it's clear to me now this is an issue I'm having with Kibana.

I just figured it out, it was an index pattern field mapping problem. Because I had old data which did not include the host metadata and just had the hostname as a string in the host field, Kibana was unable to resolve both mappings.

I recreated the index mappings to focus on more recent documents and the issue is fixed!

Glad that you posted your response back. It will be helpful for the community.

Cheers
Rashmi

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.