How to remove fields in logstash/es

Elastic Stack Logstash
1

This was my index looks like in kibana.
I don't want so many fields below, so how can i delete/avoid those from my index.
Ex.: id, _score, version, beat info.

{
  "_index": "test",
  "_type": "testing",
  "_id": "AVqK6n0wvKz7",
  "_score": null,
  "_source": {
    "@timestamp": "2017-03-01T17:27:48.830Z",
    "offset": 207,
    "@version": "1",
    "input_type": "log",
    "beat": {
      "hostname": "ip",
      "name": "ipl",
      "version": "5.0.2"
    },
    "host": "ipl",
    "source": "/var/log/ambari.log",
    "message": "ERROR [main] DBAccessorImpl:109 - Error while creating database accessororg.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.",
    "type": "error-log",
    "tags": [
      "ambari",
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ]
  },
  "fields": {
    "@timestamp": [
      1488389268830
    ]
  },
  "sort": [
    1488389268830
  ]
}
2 Likes
2

Note: Please encapsulate your code/console pasts within triple-backticks: ```. I've done this for you.

Removing them in ES is a full delete->reindex operation. Very doable, but perhaps not easy.

Really, the only things that actually come in your document are the things in _source. You can't eliminate the _index, _type, _id, and _source fields as they are ES metadata. _score is generated at search time, so it's not actually in your document.

@timestamp is necessary, and beat (and its sub keys) are used as identifiers, but can possibly be deleted. The others are perhaps up to you.

It's pretty easy to remove fields in Logstash:

filter {
  mutate { remove_field => [ "field1", "field2", "field3", ... "fieldN" ] }
}
3 Likes
3

i added in my filters, i tried both cases it doesn't work for me.
mutate { remove_field => [ "version" ] }
mutate { remove_field => [ "@version" ] }

am i missing anything ?

4

@version probably can't be removed as it is a protected field in Logstash.

1 Like
5

"_index": "test",
"_type": "testing",
"_id": "AVqK6n0wvKz7",
"_score": null,

i want to remove these fields , how can i remove those?

mutate { remove_field => [ "_id", "input_type", "[beat][version]", "[beat][name]" ] }

was not able to remove "_id" rest everything i can remove

6

Please re-read what I said above:

The fields inside of _source are the ones that come from Logstash. You cannot remove the fields outside of _source.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.