We have an application around data visualisation running purely from Elasticsearch and have a question regarding the our current data structure within Elastcsearch and how that can lead to mapping explosion.
The data structure looks like below:
As you see in the picture, this is a multi tenancy application and each index holds the data related to a particular client.
Again, as per the picture, we have data_1, data_2, data_3, etc. which holds completely different types of data which the client uses for visualisation/reporting purpose. And we don't have any control on the type of data that goes under these keys as its uploaded by the client and it differs from client to client. So we have applied dynamic mapping to each of these indices to hold any type of data (changing key value pair) as you see in the pic.
As you can see, since we have 5 types of mapping for each of these fields under data_*, the total data mapping in the index grows to a very large number (20,000 in one of our clients with 10 data sets) and we expect this to grow a lot more for bigger clients with hundreds of data sets which can lead to mapping explosion.
What should be the data structure in a scenario like this? Should we move the data sets (data_1, data_2 etc.) as indices? If so, how can we apply some other logical separation for each client as 'types' are no longer available in latest version of Elasticsearch?
Also what would be the maximum recommended value for index.mapping.total_fields.limit setting? Right now we have set it to '100000000' to accommodate big data sets.
On top of that we will have new features built frequently and the associated data has to be stored in Elasticsearch as well. We cannot afford to have a down time every time we release a new feature to change the mapping to include this new feature's data, hence we have created a key 'features' (as in the pic) with dynamic mapping like 'data_*' so that we can query the data under this 'features' key.
- Is there a better approach to minimise downtime while introducing these types of new mapping requirements?