How to remove fields in logstash/es

[bob-bza]

This was my index looks like in kibana.
I don't want so many fields below, so how can i delete/avoid those from my index.
Ex.: id, _score, version, beat info.

{
  "_index": "test",
  "_type": "testing",
  "_id": "AVqK6n0wvKz7",
  "_score": null,
  "_source": {
    "@timestamp": "2017-03-01T17:27:48.830Z",
    "offset": 207,
    "@version": "1",
    "input_type": "log",
    "beat": {
      "hostname": "ip",
      "name": "ipl",
      "version": "5.0.2"
    },
    "host": "ipl",
    "source": "/var/log/ambari.log",
    "message": "ERROR [main] DBAccessorImpl:109 - Error while creating database accessororg.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.",
    "type": "error-log",
    "tags": [
      "ambari",
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ]
  },
  "fields": {
    "@timestamp": [
      1488389268830
    ]
  },
  "sort": [
    1488389268830
  ]
}

[theuntergeek (Aaron Mildenstein)]

Note: Please encapsulate your code/console pasts within triple-backticks: ```. I've done this for you.

Removing them in ES is a full delete->reindex operation. Very doable, but perhaps not easy.

Really, the only things that actually come in your document are the things in _source. You can't eliminate the _index, _type, _id, and _source fields as they are ES metadata. _score is generated at search time, so it's not actually in your document.

@timestamp is necessary, and beat (and its sub keys) are used as identifiers, but can possibly be deleted. The others are perhaps up to you.

It's pretty easy to remove fields in Logstash:

filter {
  mutate { remove_field => [ "field1", "field2", "field3", ... "fieldN" ] }
}

[bob-bza]

@theuntergeek

filter {
mutate { remove_field => [ "field1", "field2", "field3", ... "fieldN" ] }
}

i added in my filters, i tried both cases it doesn't work for me.
mutate { remove_field => [ "version" ] }
mutate { remove_field => [ "@version" ] }

am i missing anything ?

[theuntergeek (Aaron Mildenstein)]

@version probably can't be removed as it is a protected field in Logstash.

[bob-bza]

@bob-bza

_id": "AVqK6n0wvKz7",
"_score": null,

"_index": "test",
"_type": "testing",
"_id": "AVqK6n0wvKz7",
"_score": null,

i want to remove these fields , how can i remove those?

mutate { remove_field => [ "_id", "input_type", "[beat][version]", "[beat][name]" ] }

was not able to remove "_id" rest everything i can remove

[theuntergeek (Aaron Mildenstein)]

Please re-read what I said above:

@theuntergeek

Really, the only things that actually come in your document are the things in _source. You can't eliminate the _index, _type, _id, and _source fields as they are ES metadata. _score is generated at search time, so it's not actually in your document.

The fields inside of _source are the ones that come from Logstash. You cannot remove the fields outside of _source.