First of all, regarding your comment:
APM is an incredible tool and it tracks very well
With that said, what you want to do is perfectly doable with Elastic APM — thanks to its foundation based on the Elastic Stack. You have a couple of choices:
- Design your aggregatable metric index as a data stream
- Compute an entity-based index using a transform
Let me highlight these options so you can better choose.
Design your aggregatable metric index as a data stream
Data streams behave just like plain indices, except they are the append-only indexes that can span multiple physical indexes behind the scenes. Conceptually, you will append data into something virtual. How many indexes will be created will depend on how you want to partition them — something you can implement with an ILM. Using data streams, you can accumulate your metrics as temporal-based data and run your aggregate queries to fetch data any time you want. How large the data stream may become is manageable: you will break down the indexes into smaller ones already using the ILM rules, and historical data can be sent to a cheaper storage system such as object storage using searchable snapshots.
@aravindputrevu from the Elastic community team talked about how data streams work here:
Compute an entity-based index using a transform
Instead of having an ever-growing index where you will be running point-in-time-queries to retrieve your daily aggregates, you can build an entity-centric index that will always present the metrics per day given a summarization process that can happen automatically and behind the scenes called transforms. Transforms can be executed either in batch or in a continuous mode. By continuous, I mean that as new data arrive into the source index (that can very well be a data stream), the aggregate is re-calculated to snapshot the latest results into the output index — which is going to sustain your dashboard.
The tutorial below may provide you a good understanding of how transforms works:
I think it makes perfect sense to combine the two approaches, but the best thing about the Elastic Stack is that you have options to choose from. I wrote last year a demo that uses these two concepts to build a real-time scoreboard for the Pac-Man game:
Let me know if you need more help with this. I'm glad to help you build this