Hello,
Im using elasticsearch for my application, I will explain the basic info about my app first.
We're using elasticsearch for agrregations, in every type we have 2 key fields, that we will call them A and B. They are both hierarchical string fields, A is built from numbers divided by dots, (example: 00001.00002.00004.00002). In this example, the current document father regarding the A field is 00001.00002.00004, and his father is 00001.00002 as so on. B is built as number in the similiar way (example: 24356), and his father is 2435, and his father is 243. Every document has more fields, that we aggregate thorugh and sum them using fields A and B. for example: I have for each document the fields C, D and F that are integer. If i want to know the value the combination of A = 00001. and B = 20, I will make and aggregation with prefixes (all the document that their A field starts with 00001 and their B fields starts with 20), and then I do the sum of this group i filtered for field C, D and F. In this way, when i want to know the info about an object A = 00001 it will sum all the documents that start with 00001 (00001.00003, 00001.00002, 00001.00001.00006), ans thats how I use elasticsearch for aggregation in my app.
Now I also want to save the data in point of times at the past, so if the user want to know how much an object had of field C in monday of last week at 9PM, he can get that info, What im currently doing is saving in each month the values of all the documents in a different delta's index, and than saving each delta in information with the timestamp of the change (example: If document had value of field C = 40 at the start of the month, later that month we changed value to 45, so i will index to my delta's index a new document where the value of field C is 5 (the amount changed). That's not good and I want to change it, it's realy not safe a it gives us a lot of problem in my app.
The things I want to do, is that instead of saving the delta, save the actual value. The problem is that I dont know how can I get that info when i need it. for example:
If i need the the valued of field C, D, F for the object that his A field is 00001, and his B field is 20 at the 26.08.15 10:30:00. So what I need to do is to filter documents based on the prefix of A and b, and also filter the document so I will only want fields that were indexed before 26.08.15 10:30:00 (using timestamp), and then I need to get the documents with the max timestamp, because if I have 2 document s with the same primary key (A and B are the same) I want to get the last value of this document because thats the document that is up to date. and after I get a group of only the documents I want, I need to aggregate through them and sum the fields C, D, and F.
I dont know how to do this or if you can even do it. And if not, Im looking for another way to save the history of my documents.
Sorry for the long post 