# Is it possible for ES to drop every field that's not in mapping upon indexing

**URL:** <https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449>\
**Category:** Elasticsearch\
**Created:** [July 12, 2021, 5:50pm UTC](https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449 "2021-07-12T17:50:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![R.L](https://avatars.discourse-cdn.com/v4/letter/r/22d042/32.png) [@R.L](https://discuss.elastic.co/u/R.L)\
**Post date:** [July 12, 2021, 5:50pm UTC](https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449/1 "2021-07-12T17:50:16Z")

</div>

Let's say I have a index template mapping which has fields:

```auto
{
 "first_name"...,
 "last_name"...,
}

```

and someone is uploading docs with fields:

```auto
{
   "first_name": "jon",
   "last_name": "connington",
   "address": "blah"
}

```

Is there an option or some way so ES automatically just don't include "address": "blah" in \_source upon indexing the doc?

I know it's possible for those fields to not be searchable. But clients talk about scalability from storage perspective and don't want to store extra fields to avoid extra costs.

I'm currently using Logstash to pipe data to ES, I know I can use remove\_fields or prune. But there's a lot of different metric types and I would rather not have to manually remove fields for every single type if possible.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 12, 2021, 6:40pm UTC](https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449/2 "2021-07-12T18:40:41Z")

</div>

Perhaps see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html)

Set dynamic mappings to false, you can have granular level control as well

```auto
PUT my-index-000001
{
  "mappings": {
    "dynamic": false, 
    "properties": {
      "user": { 
        "properties": {
          "name": {
            "type": "text"
          },
          "social_networks": {
            "dynamic": true, 
            "properties": {}
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![R.L](https://avatars.discourse-cdn.com/v4/letter/r/22d042/32.png) [@R.L](https://discuss.elastic.co/u/R.L)\
**Post date:** [July 12, 2021, 6:48pm UTC](https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449/3 "2021-07-12T18:48:40Z")

</div>

Thank you for the reply, however, I see this part of the doc:

> `false` New fields are ignored. These fields will not be indexed or searchable, but will still appear in the `_source` field of returned hits. These fields will not be added to the mapping, and new fields must be added explicitly.

My objective is so those fields are never in the " \_source" field in the first place in order to conserve storage space.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 12, 2021, 6:55pm UTC](https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449/4 "2021-07-12T18:55:21Z")

</div>

Ahhh....

AFAIK You will need to do some processing etc before hand `_source` is the data before the document is indexed, and only when the document is indexed is it compared to the mapping and that process does not affect / edit the `_source`.

I am thinking that, you will need to do something like an ingest / logstash pipeline "pre indexing" to filter / remove those fields from the `_source` (the source document)

Perhaps someone else will have an alternate / crafty method.

---

<div class="post-metadata">

**Author:** ![R.L](https://avatars.discourse-cdn.com/v4/letter/r/22d042/32.png) [@R.L](https://discuss.elastic.co/u/R.L)\
**Post date:** [July 12, 2021, 6:58pm UTC](https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449/5 "2021-07-12T18:58:52Z")

</div>

yeah "pre-processing" with remove\_fields based on ES templates is what I'm doing now. It's just tedious.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 9, 2021, 6:59pm UTC](https://discuss.elastic.co/t/is-it-possible-for-es-to-drop-every-field-thats-not-in-mapping-upon-indexing/278449/6 "2021-08-09T18:59:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.