Aggregating rate values over time

Elastic Stack Kibana
1

I've got a Kibana chart that is displaying a rate, say "number of records processed/second" from a data source. I can have several hosts that are processing data. If I aggregate these using the "sum" method, then the values display correctly even if I filter some of the hosts out of the chart. But summing these rates over time does not provide meaningful data, since adding the rate for now and the rate five minutes ago is not meaningful.

My problem is that my interval is ">=1m". We have data points for every minute, but if I specify the interval for one minute, my chart can only hold 2000 data points. If I make the chart's timeline too long, the data cannot be displayed. If I set the aggregation to "max" or "avg", Kibana will combine data points and display the "max" or "avg" of the data for 5 minutes/an hour/a day depending on the length of time displayed on the chart. However, "max" only displays one of the hosts, and "avg" gives a meaningless value since some hosts usually have a rate of zero.

Is it possible for my Kibana visualization to "sum" each data point for a particular minute, but only display the maximum when data points are combined due to a long timeline? Am I stuck with forcing the interval to be "1min"?

2

Is there a way to get the size of the interval displayed on a chart when the interval is ">=1m"? If I knew that I could write a painless script to get the average of the data points.

3

image
image.png1725×344 17.2 KB

I've found a partial solution, but it doesn't seem to work completely right.

Displaying the "max" value for each data point/time period, and then aggregating the "sum" of these will only display correctly if I group by "terms" and display the max of my values. For some reason, grouping by "Everything" only displays one maximum value, and does not aggregate the maximum values. Even if my line chart is "stacked", I only get one value when I group by "terms". I thought that the top "n" terms were supposed to be graphed.

Is this a bug in Kibana?

4

We were upgrading from Kibana 6.2.4 to 6.7.1 anyway, so I've tried this same chart in 6.7.1. However, everything still does not mean "everything". I still have to group by "terms" and display the top x values, but I only get one line which is the same as everything.