Drop_fields and the beats.name/.hostname/.version fields

Elastic Stack Beats
1

Hi There,

I'm using Metricbeat, Elastic Search and Kibana to create system overview dashboards of about 45 Linux boxes.

I'm trying to keep the data to the necessities, so I'm stripping out as much data as possible using drop_fields.

I'm using these filters on the systems module in :

- drop_fields:
       fields: [ "beat.version", "metricset.module", "metricset.rtt", "host", "system.load.norm.5", "system.load.5", "system.load.norm.15", "system.load.15", "system.load.norm.1"

However I can't find a way to get rid of the 'beat' fields, I've tried various things like this:

*beat* 
beat.* 
beat.hostname 
*.hostname

Equally, has anyone any idea if I can do anything with the _ fields? I suspect these are required, I'm just trying to shave anything I can to keep costs down and performance on the elasticsearch box up :slight_smile:

{
  "_index": "metricbeat-2017.12.12",
  "_type": "logs",
  "_id": "AWBMlbcud7P8UCMuIZjH",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2017-12-12T21:16:07.321Z",
    "system": {
      "load": {
        "1": 0,
        "cores": 2,
        "norm": {
          "1": 0
        }
      }
    },
    "beat": {
      "name": "supportserver",
      "hostname": "supportserver",
      "version": "6.0.1"
    },
    "@version": "1",
    "host": "supportserver",
    "metricset": {
      "name": "load"
    },
    "tags": [
      "beats_input_raw_event"
    ]
  },
  "fields": {
    "@timestamp": [
      1513113367321
    ]
  },
  "highlight": {
    "metricset.name": [
      "@kibana-highlighted-field@load@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1513113367321
  ]
}

Thanks in advance for any help :slight_smile:
Mike

2

Does it work if you use this?

- drop_fields:
    fields: [beat]

Those are not sent by Beats. They are part of Elasticsearch. Index API | Elasticsearch Guide [8.11] | Elastic

3

Trying just 'beat' doesn't work I'm afraid. Tried with and without quotes, currently looks like this in my config:

  - drop_fields:
      fields: [ beat, "system.beat.name", "*name*", "beat.version", "metricset.module", "metricset.rtt", "host", "system.load.norm.5", "system.load.5", "system.load.norm.15", "system.load.15", "system.load.norm.1"]

Thanks for your reply :slight_smile:

4

Try putting into metricbeat.yml

processors:
- drop_fields:
    fields: [beat]

This works for me when setting it at the global level on Metricbeat 6.0.1.

It didn't work at the module level. I assume this is because the beat object isn't added until later in the pipeline.

5

Yup! That did the trick.

Oddley, I've lost 'host' even after having removed it from my module sided filters, but I think I can handle that with one tag per host anyway.

Thanks for your help!

P.s: Is there some way I can contribute to the documentation? I feel like this should be mentioned as a footnote on the drop_fields section on the individual module pages.

6

It's all open source and contained in GitHub - elastic/beats: 🐠 Beats - Lightweight shippers for Elasticsearch & Logstash so you can open a PR with suggested edits. From the docs on the elastic.co site there are edit links on the right side of every section. Clicking that link takes you directly to github web editor for the page (if you are signed into github).

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.