Reindexing a document with same version number but with different metadata

Elastic Stack Elasticsearch
1

Hi,

I have the following use case.

  1. An admin indexes the existing system during a small maintenance window , all good to go. We plan to use an external version for index document version
  2. Users starts using the system again.. Live changes are indexed as well using the versions that exists in our database. Any requests that happens within millisec difference and goes to ES in a different order will get rejected as version numbers are not in order.. All good.
  3. Now let us say doc1 is v3 . Now our system is configurable in a way that an admin could define , there are more business properties that needs to be part of the index. doc1 , v3 in our system has all this business fields, just that they were not configured/chosen as data that needs to be indexed.
  4. Now the admin operation in 3, has to trigger an index on doc1 v3 again, but this time with more data.. If use external or external_gt , that operation will fail as version number is there
  5. If I dont choose versioning at all, scenario in point 2 above will have issues in a highly transactional system.

I thought this is a scenario which is good for usage of external_gte version type. But the documentation warns about using that version type.. What are the potential issues if I use that option ?

Another option is to delete v3 , wait for 1 minute ( or configurable index.gc_deletes seconds) , and then reindex again... Not sure about the performance impacts of reducing index_gc_deletes to say 5 seconds..

Any suggestions or thoughts on how best the above scenario can be handled in ES ?

2

reason for choosing to index only configured metadata ( from doc1 v3 in above example) instead of indexing all metadata is to avoid the hard limits on number of metadata properties that can be indexed in a given index

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.