I've been testing the rollup feature and I like it, a lot, cause it is something we wanted to do manually but never found the time to implement it. It is simple, easy and can save tons of space. The problem is that it is very limited.
Let's say we have API requests on ElasticSearch, and we have a "status" field with the HTTP response status code. I can configure the rollup job to include this field for terms aggregation but I find it frustrating that I cannot query the rollup index with q="status:200" for example. I mean, the data is there! Why can't I filter?
Ok ok, I know it is in beta, and I know it is tricky. Cause at some point I will want to query by status and URL too, for example. And storing all the field combinations in a single rollup index would not scale well I guess.
I have an idea I would like to share with you, please let me know if it makes any sense or not:
Instead of filtering data after the rollup job has aggregated everything, can we filter it when doing the rollup? Is this very difficult to implement?
It would be very useful, at least for me. I could have a rollup for errors, a global rollup for all traffic, maybe a rollup for a specific endpoint I want to track over time, or a rollup for requests that take longer than X ms to respond... Much more powerful, don't know what you think.
Anyways, thanks for reading and thanks for the rollup feature, I think it has a lot of potential!
