Keeping latest document only

Elastic Stack Elasticsearch
1

in My application Every day 6 times (4hrs per run) index will be updated by latest exchange rates documents.
Any one could guide me on best approach how to delete my old documents (having SourceCurrency=USD) from logstash and insert new documents to elasticsearch (for SourceCurrency=USD)?

  1. I have multiple documents for same SourceCurrency=USD.
    Capture

  2. I dont want to update price document. I have to delete all old price documents having (SourceCurrency=USD) and then new document should be inserted ( since every run number of documents will vary for same SourceCurrency=USD)

  3. there is no document_id i havent defined in index. i cant create document id with combination of SourceCurrency+DestinationCurrency since i may not sure will get same number of destination currency all time. i dont want to retain the destination currency which is not received in latest run.

2

Hello,

You should create a new index each time.

Then, instead of searching against an index name, you can search against an alias that "points" to the most recently completed index.

As soon as you complete a new index job, you'll issue a command to remove the old index and add the new one to the alias (you can do both in one command so it's an atomic change), and your searches will now use the new index, and you can delete the entire old index.

This approach is preferable for operational simplicity, as well as the fact that document updates have deletions underneath, and a large number of deletions results in increased resource usage for the segment merge operations; deleting the entire old index is a simple filesystem level operation.

3

Thanks Glen but one of my usecase not having fixed interval re-indexing - which means - there is no time specific indexing job, as long as data come in (irrelevant interval ) to elastic search i need to start indexing it and make it available for consumer search in that case i would like to only replace the currency exchange json document which is impacted. do i need to handle using primary key concepts?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.