This is a common warning seen due to what is commonly known as a mapping explosion. The docs go into this is more detail, but quickly (emphasis by the author);
Defining too many fields in an index can lead to a mapping explosion, which can cause out of memory errors and difficult situations to recover from.
Consider a situation where every new document inserted introduces new fields, such as with dynamic mapping. Each new field is added to the index mapping, which can become a problem as the mapping grows.
Below contains 4 potential solutions, with some explanation on each.
Be explicit with your mapping
Static mappings differ from dynamic in that one of three things can happen to any document that is submitted with fields that are not explicitly mapped;
- By default, when a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping
- Setting
"dynamic": "false"means the documents will be accepted, but those extra fields will be ignored (that is, dropped by Elasticsearch) - Setting
"dynamic": "strict"will reject the document entirely, and pass an exception back to the client if an unknown field is encountered
The docs have more on this topic, explicit mapping here and the dynamic parameter here.
Use a flattened structure
If you have a lot of custom fields, it may make sense to use something like a flattened structure to reduce complexity. There may be some added work in your client to read these, but it provides a lot of flexibility.
Split your index
This might be the result of both of of the above changes, where altering your mappings means that [N] previously similar docs have enough differences to split them out.
The benefit here is that you can then reindex older data into the new index and mapping, optimising your overall index size. If you take this approach, it might also be worth using ILM to manage your indices, which will reduce concerns about over sharding.
Increase index.mapping.total_fields.limit
This should only be considered a temporary solution until you can deploy one of the others above, as otherwise it becomes a permanent "oh, just increase it again" situation. It can be applied dynamically to an existing index, or assigned at index creation time. The default value is 1000.
PUT my_index/_settings
{
"index.mapping.total_fields.limit": 2000
}