# Remove unnecessary fields in ElasticSearch

**URL:** <https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673>\
**Category:** Logstash\
**Created:** [September 21, 2015, 5:37am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673 "2015-09-21T05:37:33Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hayder\_Abbass](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hayder_abbass/32/4559_2.png) [@Hayder\_Abbass](https://discuss.elastic.co/u/Hayder_Abbass)\
**Post date:** [September 21, 2015, 5:37am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/1 "2015-09-21T05:37:33Z")

</div>

Hello,

We are populating Elasticsearch via logstash. The thing is that I see some unnecessary fields that I had like to remove like for example:

```
@version
file
geoip
host
message
offset
tags

```

Is it possible to do this by defining/extending a dynamic template? If yes, how? If no, can we do this via logstash configuration?

Your help is much appreciated.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2015, 5:41am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/2 "2015-09-21T05:41:47Z")

</div>

Best to do this in LS, I can move the topic there if you want?

---

<div class="post-metadata">

**Author:** ![Hayder\_Abbass](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hayder_abbass/32/4559_2.png) [@Hayder\_Abbass](https://discuss.elastic.co/u/Hayder_Abbass)\
**Post date:** [September 21, 2015, 5:43am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/3 "2015-09-21T05:43:02Z")

</div>

Yes, please 🙂

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2015, 5:44am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/4 "2015-09-21T05:44:37Z")

</div>

Ok, so you can remove the fields using the mutate filter.

Are these dynamic fields, or static?

---

<div class="post-metadata">

**Author:** ![Hayder\_Abbass](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hayder_abbass/32/4559_2.png) [@Hayder\_Abbass](https://discuss.elastic.co/u/Hayder_Abbass)\
**Post date:** [September 21, 2015, 5:46am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/5 "2015-09-21T05:46:31Z")

</div>

They are being created by Logstash. So I guess they are dynamic.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2015, 5:47am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/6 "2015-09-21T05:47:58Z")

</div>

Hmm, ok then it might actually be better to do this in ES via mapping and then just drop anything that doesn't fit your structure. Do you know exactly what you want to keep?

---

<div class="post-metadata">

**Author:** ![Hayder\_Abbass](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hayder_abbass/32/4559_2.png) [@Hayder\_Abbass](https://discuss.elastic.co/u/Hayder_Abbass)\
**Post date:** [September 21, 2015, 5:55am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/7 "2015-09-21T05:55:15Z")

</div>

Yes, that's my complete structure:

```
{
   "logstash-2015.09.15": {
      "mappings": {
         "ufdb": {
            "dynamic_templates": [
               {
                  "message_field": {
                     "mapping": {
                        "type": "string"
                     },
                     "match": "message",
                     "match_mapping_type": "string"
                  }
               },
               {
                  "string_fields": {
                     "mapping": {
     
                        "type": "string",
                        "fields": {
                           "raw": {
                              "type": "string"
                           }
                        }
                     },
                     "match": "*",
                     "match_mapping_type": "string"
                  }
               }
            ],
            "_all": {
               "enabled": true,
               "omit_norms": true
            },
            "properties": {
               "@timestamp": {
                  "type": "date",
                  "format": "dateOptionalTime"
               },
               "@version": {
                  "type": "string"
               },
               "category": {
                  "type": "string" ,
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "clientip": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "file": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "geoip": {
                  "dynamic": "true"
               },
               "group": {
                  "type": "string",
                  "fields": {
                     "raw": {
                        "type": "string",
                        "index": "not_analyzed",
                        "ignore_above": 256
                     }
                  }
               },
               "host": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "logdate": {
                  "type": "date",
                  "format": "dateOptionalTime"
               },
               "message": {
                  "type": "string"
               },
               "method": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "offset": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "status": {
                  "type": "string"
                  "fields": {
                     "raw": {
                        "type": "string"
                     }
                  }
               },
               "tags": {
                  "type": "string"
           
               },
               "type": {
                  "type": "string"
               },
               "url": {
                  "type": "string"
               }
            }
         }
      }
   }
}

```

I want to keep everything except the fields mentioned in my original message above. Thanks for your help.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2015, 10:32pm UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/8 "2015-09-21T22:32:28Z")

</div>

> [@Hayder\_Abbass](#):
>
> @version  
> file  
> geoip  
> host  
> message  
> offset  
> tags

Ok, then if it's only those just use a mutate + remove\_field as per [Mutate filter plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field).

---

<div class="post-metadata">

**Author:** ![yahoo](https://avatars.discourse-cdn.com/v4/letter/y/898d66/32.png) [@yahoo](https://discuss.elastic.co/u/yahoo)\
**Post date:** [February 11, 2016, 12:10am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/9 "2016-02-11T00:10:40Z")

</div>

Hi @warkolm  
If I understand you correctly,  
in order to tailor the discover part of Kibana for a particular presentation, the data should not be even permitted to enter the system.  
This data should be removed right at the entrance via logstash.

Thus, in order to provide custom presentations for the same data in Kabana I have to run a separate (logstah, elasticsearch, kibana) stack per presentation?

---

<div class="post-metadata">

**Author:** ![yahoo](https://avatars.discourse-cdn.com/v4/letter/y/898d66/32.png) [@yahoo](https://discuss.elastic.co/u/yahoo)\
**Post date:** [February 11, 2016, 1:03am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/10 "2016-02-11T01:03:22Z")

</div>

Sorry.  
To be more specific.

In order to provide custom presentations, in terms of the fields shown in the discover part of Kibana, I have to run a separate ELK stack per presentation?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 11, 2016, 2:45am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/11 "2016-02-11T02:45:00Z")

</div>

It's probably better if you create your own thread for that question.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:12am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673/12 "2017-07-06T05:12:08Z")

</div>