Limiting dynamic properties/fields indexed

Elastic Stack Elasticsearch
1

We're migrating to ES7, and for one of our indexes which stores email data we are hitting the "Limit of total fields [1000]" error.

The reason for this is we're indexing all headers, and for some emails the number of headers in addition to the standard fields exceeds 1000.

What I'd prefer to do is set up the headers field so that ES simply ignores any properties received over a number I configure. So for example, I'd want to limit the number of headers to the first 100 that ES encounters and have it quietly ignore the rest, maybe logging a warning in the task.

For future data we could restrict the number of fields at index time, but we want to be able to reindex existing documents from another cluster without regenerating all the documents - which is an expensive task.

I haven't been able to find anything that allows me to configure a particular property that limits the fields beneath it. Is there anything like that?

2

There is no setting to limit the number of fields under a specific field, there is only index.mapping.total_fields.limit that limits the number of fields for the whole index.

3

That's what I figured, we'll look to adjust that number up higher by default but also limit the number of fields we index to keep it running more robust

4

One thing probably worth to add is that you can set up dynamic: false on the specific object field level. This will allow to add new fields to your index, but will not allow to add new field under the given object.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.