Remove unnecessary fields in ElasticSearch

Elastic Stack Logstash
1

Hello,

We are populating Elasticsearch via logstash. The thing is that I see some unnecessary fields that I had like to remove like for example:

@version
file
geoip
host
message
offset
tags

Is it possible to do this by defining/extending a dynamic template? If yes, how? If no, can we do this via logstash configuration?

Your help is much appreciated.

2

Best to do this in LS, I can move the topic there if you want?

1 Like
3

Yes, please :slight_smile:

4

Ok, so you can remove the fields using the mutate filter.

Are these dynamic fields, or static?

5

They are being created by Logstash. So I guess they are dynamic.

6

Hmm, ok then it might actually be better to do this in ES via mapping and then just drop anything that doesn't fit your structure. Do you know exactly what you want to keep?

7

Yes, that's my complete structure:

{
   "logstash-2015.09.15": {
      "mappings": {
         "ufdb": {
            "dynamic_templates": [
               {
                  "message_field": {
                     "mapping": {
                        "type": "string"
                     },
                     "match": "message",
                     "match_mapping_type": "string"
                  }
               },
               {
                  "string_fields": {
                     "mapping": {
     
                        "type": "string",
                        "fields": {
                           "raw": {
                              "type": "string"
                           }
                        }
                     },
                     "match": "*",
                     "match_mapping_type": "string"
                  }
               }
            ],
            "_all": {
               "enabled": true,
               "omit_norms": true
            },
            "properties": {
               "@timestamp": {
                  "type": "date",
                  "format": "dateOptionalTime"
               },
               "@version": {
                  "type": "string"
               },
               "category": {
                  "type": "string" ,
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "clientip": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "file": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "geoip": {
                  "dynamic": "true"
               },
               "group": {
                  "type": "string",
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "host": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "logdate": {
                  "type": "date",
                  "format": "dateOptionalTime"
               },
               "message": {
                  "type": "string"
               },
               "method": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "offset": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "status": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "tags": {
                  "type": "string"
           
               },
               "type": {
                  "type": "string"
               },
               "url": {
                  "type": "string"
               }
            }
         }
      }
   }
}

I want to keep everything except the fields mentioned in my original message above. Thanks for your help.

8

Ok, then if it's only those just use a mutate + remove_field as per Mutate filter plugin | Logstash Reference [8.11] | Elastic.

9

Hi @warkolm
If I understand you correctly,
in order to tailor the discover part of Kibana for a particular presentation, the data should not be even permitted to enter the system.
This data should be removed right at the entrance via logstash.

Thus, in order to provide custom presentations for the same data in Kabana I have to run a separate (logstah, elasticsearch, kibana) stack per presentation?

10

Sorry.
To be more specific.

In order to provide custom presentations, in terms of the fields shown in the discover part of Kibana, I have to run a separate ELK stack per presentation?

11

It's probably better if you create your own thread for that question.