I am wondering if someone else has faced a similar use case that I have and if so how did they handle it. I have a master list of data say it has 100 rows with unique IDs, over a period of time those 100 will change. But not all 100 will change always, 10 may change today another 20 may change tomorrow. I am sending a -seed index with all 100 to elastic and then sending the changed records daily. However, I need to maintain the "state of the last record" which means if one of those 100 records did not change, then the "last known value" should be displayed. This value actually needs to be added into aggregations as well. Is there a way to do this in elastic? Basically something like "version the data" and use the latest version. So out of those 100, one record might be from today but one might be from a year ago since it never changed in that year! Ofcourse the index wont have and updated record for the one that never changed, but how would you consider this in a visualization.
Do you need to see earlier versions?
If not then just make your unique ID the choice of _id for the documents and new content can be made to update existing docs or create if missing.
I kinda do need the earlier data. Today we just update the index with the updated data, however I need to keep the history. However in certain dashboards, only the most recent data for that account would make sense.
Then keep 2 sets of indices: one where you store all history and one where you update to always only have the latest version. Then query the index that is most appropriate.
OK. Thats what I thought! Thanks.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.