Cumulative graph

Elastic Stack Kibana
1

Hi,

I am trying to find an efficient way to display the growth of a number of a type of requests over a period of time.
I managed to create a graph (see below) that shows what we want, but it needs to have indices from the beginning of the specific project, which means that we need to keep a lot of indices just for counting those requests, when we actually need logs only for few weeks back.

Pasted image1737×975 44.1 KB

I have tried using a watcher that would query all requests of that type, but I was unable to get the result.input.payload.hits.total in a visualisation.

Would a scenario like this be possible:

  1. Set a watcher to get the count of all requests
  2. Edit the watcher to get the total hits and date from the previous execution and then somehow return the sum of the total hits and hits from that date to now
  3. Make a graph using the sum from the watcher?

This way we could cleanup old indices that we don't need and use the watcher index instead.

2

I think you're asking if you can somehow "roll up" your data to just extract the counts for you, instead of keeping the actual hits and using aggregations to count them. Is that right?

I know there are tools available for rolling up data, but I couldn't tell you what any of them are. I also am not sure if this is something you can do with Watcher. You might try asking over on the Elasticsearch section.

The use case you are describing is exactly what Elasticsearch's aggregations are doing on the fly though. Is there a reason you need to roll this data up? It doesn't look like you're dealing with a lot of log records, at least based on that visualization you've shown here. It's going to be more work to roll it all up, and if you ever want to change what you look at, like if you wanted to view that data daily or hourly instead of weekly, you have to do it all over again.