Dealing with changes in field datatype

Elastic Stack Elasticsearch
1

Hello,

I have an Elasticsearch index where fields are configured by clients (indirectly) i.e., the clients configure the datatype for these fields through an interface and then I have a workflow which creates the actual mapping for these fields in the index. The clients then index data and perform queries.

If a client wishes to change the field datatype for a given field, I would have to re-index all documents to a new index to change the datatype for the field. This is fine if performed infrequently, however if some client changes the datatype too many times, too many reindexing operations are not desirable, given the fact that the index contains a lot of documents from other clients as well (greater than 30 million and growing) and I feel this would affect the cluster negatively.

Is there a way where I can solve this without having to re-index frequently? Would field aliases work here? Eg: I create a field alias for a field that client wishes to create. Every time client changes the datatype, I would just create a new field and point the field alias to the new field.
Any other workarounds/suggestions are also welcome.

Thanks,
Raonic

2

Yeah that seems to be the best approach, given your other limitations.

3

However, I just realized field aliases cannot be used in index or update operations. So I wonder how writes can be performed.
Example:

  1. User creates field postalCode of type text. The workflow creates the alias postalCode and points it to a field called postalCode-0001 of type text.
  2. Now, user changes the type of the field to type long. The workflow simply creates a new field postalCode-0002 of type long and changes the alias to point to this new field.
  3. How do I manage write operations here? Since I cannot use the postalCode alias field to perform the writes.

Thanks,
Raonic

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.