Just to add to the discussion, I've recently been looking into options for doing calculations across events/documents. I've come across the following options, although I don't have a wealth of direct experience in any of them.
- Logstash Aggregate filter - Can piece together related events and saves a single event after the final event has been detected (or timed out). Has some scaling concerns.
- Logstash Elasticsearch filter and elasticsearch output - Logstash can query elasticsearch for a previously-ingested event/document, use its fields to calculate something new, and then update the original document.
- Elasticsearch Transforms - After events have been ingested, transform them into an entity-centric index using the Transforms feature. I'm not sure how much delay you can expect from this post-processing, but Transforms can run in continuous mode, so in theory it can be fairly minimal.
Anyone, feel free to correct me on anything. There's also some good discussion here: