Update and snapshot for indices with timestamp

Elastic Stack Elasticsearch
1

Hello,

I have a question about implementing update and snapshot for indices with timestamp.
The index format looks like result-2018.07 result-2018-08 which is good to use curator to do snapshot and delete. However, the documents in each indices need to be updated very frequently, say one update per minute. Since update API does not allow wildcard index, and the sender does not know exactly which index to update, what is the best design in this case?

I have googled a little bit and there are two options:

  1. just use single index, which makes the the operation very heavy.
  2. update has an option, create if not exists.

But I am also worry about them:
for option 1: if snapshot is taking the difference, and one document had been updated between two snapshots. If I restore these two snapshots, will the document appear twice?
for option 2: there could be a case that the same ID shows up in multiple indices, if I do search by id in all indices, would it give me all documents with that ID?

Thank you

2

This case is different from what you are probably expecting.

Snapshots are incremental at the segment level, not the document level, or the index level. What this means is that you can always snapshot all of your indices, and only the segments which are new or have changed since the last snapshot will be backed up. Segments which have already been copied to the repository will simply be referred to by pointer. Segments in the repository will not be deleted until all pointers/references have been removed.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.