Is it possible to trigger scripts or events when an index receives an update/insert?
For example, I have a document coming into an index with a list of values, like so:
{"_id": "A", "nodes": [1, 2, 3]}
{"_id": "B", "nodes": [2, 3]}
{"_id": "C", "nodes": [1,3]}
This is a simplification of the data--in reality, the data has a lot more values and there are other pieces of data at the top level. However, I'd like to be able to ask ES, "List the number of documents assigned to each node".
What I'd like to be able to do (for speed purposes) is keep a separate index that ONLY keeps track of the counts. For the example above, I'd like another index that has:
{"_id": 1, count: 2}
{"_id": 2, count: 2}
{"_id": 3, count: 3}
What I'd like to do is, when I receive a new document, I'd like to update these counts. So let's say "A" receives an update:
original: {"_id": "A", "nodes": [1, 2, 3]}
new: {"_id": "A", "nodes": [1, 3]}
...is there a way for me to compare the old and new, and then update the "counts" table accordingly?
(terms aggregations would work, but they are so big that they crash the cluster with 2.8 billion documents, and around 2.9 million unique terms)