Hi,
I have many millions of documents representing thousands of different metrics.
Currently, we use bulk update requests that increment all counters in the document if it is already present.
Using Update instead of indexing a new document has its performance hit during the update process (due to fetching the previous doc, updating and then deleting the old doc), so I thought about switching to append the only mode.
When querying, I have no problem handling either of the above cases, but I'm concerned about performance during the query and aggregations.
e.g
If I currently have the following documents:
{myid: 'aaa', counter1:1, counter2:4} {myid: 'bbb', counter1:3, counter2:3}
After updating with {myid: 'aaa', counter1:2, counter2:0} I will end up with:
{myid: 'aaa', counter1:3, counter2:4} {myid: 'bbb', counter1:3, counter2:3}
While the append only approach will leave me with:
{myid: 'aaa', counter1:1, counter2:4} {myid: 'bbb', counter1:3, counter2:3} {myid: 'aaa', counter1:2, counter2:0}
My queries are mainly: show sum of counters for 'aaa'\counter1
Thanks all!