Is it possible for ES to drop every field that's not in mapping upon indexing

Elastic Stack Elasticsearch
1

Let's say I have a index template mapping which has fields:

{
 "first_name"...,
 "last_name"...,
}

and someone is uploading docs with fields:

{
   "first_name": "jon",
   "last_name": "connington",
   "address": "blah"
}

Is there an option or some way so ES automatically just don't include "address": "blah" in _source upon indexing the doc?

I know it's possible for those fields to not be searchable. But clients talk about scalability from storage perspective and don't want to store extra fields to avoid extra costs.

I'm currently using Logstash to pipe data to ES, I know I can use remove_fields or prune. But there's a lot of different metric types and I would rather not have to manually remove fields for every single type if possible.

2

Perhaps see here

Set dynamic mappings to false, you can have granular level control as well

PUT my-index-000001
{
  "mappings": {
    "dynamic": false, 
    "properties": {
      "user": { 
        "properties": {
          "name": {
            "type": "text"
          },
          "social_networks": {
            "dynamic": true, 
            "properties": {}
          }
        }
      }
    }
  }
}
3

Thank you for the reply, however, I see this part of the doc:

false New fields are ignored. These fields will not be indexed or searchable, but will still appear in the _source field of returned hits. These fields will not be added to the mapping, and new fields must be added explicitly.

My objective is so those fields are never in the " _source" field in the first place in order to conserve storage space.

4

Ahhh....

AFAIK You will need to do some processing etc before hand _source is the data before the document is indexed, and only when the document is indexed is it compared to the mapping and that process does not affect / edit the _source.

I am thinking that, you will need to do something like an ingest / logstash pipeline "pre indexing" to filter / remove those fields from the _source (the source document)

Perhaps someone else will have an alternate / crafty method.

5

yeah "pre-processing" with remove_fields based on ES templates is what I'm doing now. It's just tedious.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.