Deletion on fields within an index

Elastic Stack Elasticsearch
1

We have this usecase:

  1. Create Application with fields A (text), B (text), C (text). Application is more of a UI term but we have say for each application a collection in mongoDB to store the metadata of the application.
  2. Create some records. records are like instances of the application created where we can put values for each of these A, B, C text fields. This is stored in a collection named Records.
  3. We are ingesting this records collection data to elastic for search capability.
    Expectation at this stage is that Elastic will have A, B, C in records index for search for all inserted records
  4. Delete field C in the Application Builder.
  5. Expectation is that only A and B will be there for search in Elastic.
    Step 5 is the challenge, as we need to find a way to delete/invalidate field C from all existing records in Elastic. What is the best way to achieve this? Elasticsearch will still retain that field unless you explicitly reindex them to remove it??
2

Yes. You will need to reindex or run an update-by-query to remove it.

3

Thanks @Christian_Dahlqvist for the answer. Maybe the reindex is a nightly operation to clean up elastic? How heavy of an operation is that?

4

That depends of the amount of data and performance of the cluster so you need to test it.

5

the reindex operation can be scheduled or can be done anytime?

6

You need to create a mechanism, e.g an external script, to call the reindex API or possibly the update by query API with the correct parameters. There is nothing within Elasticsearch that supports scheduling this.