Maximum number of fields in an index mapping

There are three main reasons:

Cluster state overhead: the mapping for each index is stored in the cluster state, which is shared among all nodes. Any change to the cluster state (such as adding a new field) causes the CS to update across all nodes. Very large mappings induces a non-negligible amount of data that needs to be sent over the wire and refreshed on all the nodes. It might seem unimportant, but having to periodically serialize a few mb of cluster state really adds up over time, and can add unwanted latency to regular actions.

Sparsity: Generally, people who have thousands of fields also tend to have very sparse fields. E.g. each document only has a handful of the thousand fields. This makes the data-structures stored on disk very inefficient (less so in newer versions, but still not ideal) because the data is so sparse. You tend to see this kind of behavior when ES is used as a blind key:value store, or where multiple tenants share the same index and are allowed to create whatever fields they want.

Lucene overhead: in short, having thousands of fields eats up a certain amount of fixed overhead

The limit is a soft-limit, so you can change it if you want. But it's there for a reason, namely that we think >1000 fields is starting to get abusive and we'd recommend trying to pare down your fields with some kind of alternate scheme. :slight_smile:

3 Likes