Difference of aggregated value that grouped by two fields across time

[amylyu]

Hi, we are getting data into elastic through Kafka Prometheus endpoint and trying to create a visualization which would help us show latest ingestion rate in each kafka topics. In order to achieve this, we need to group by two fields – partitions & topic name. Since this value is incremental, we also need to calculate the difference between the value we are getting now – aggregated value we had one minute before. The purpose of this calculation is to get realtime data ingestion rate/topic. Is there a way in kibana to create this kind of visualization? Thank you in advance!

We are using Elasticsearch version 7.16.

[Marius_Dragomir]

The solution for the ingest rate is using the "Serial difference" aggregation in the classic aggregation types and TSVB, or the "Differences" formula in Lens. And the grouping is available everywhere, just add a second one after the first one.

[amylyu]

Thanks for replying! I have tried using the TSVB, but I can only group by one terms. There is no way I can add the second "group by".

[Marius_Dragomir]

Click the + button next to the "By" where you select the field. That should add another selector the second field.

[Marius_Dragomir]

[amylyu]

There is no + button. Is this due to the version of Elasticsearch?

[Marius_Dragomir]

it could be, it's been added in 8.2.0. you can upgrade or use Lens instead.

[amylyu]

If I use Lens by the line chart, how can I aggregate by two terms? I can break down by one field, but I failed to break down by another one.