Hi, we are getting data into elastic through Kafka Prometheus endpoint and trying to create a visualization which would help us show latest ingestion rate in each kafka topics. In order to achieve this, we need to group by two fields – partitions & topic name. Since this value is incremental, we also need to calculate the difference between the value we are getting now – aggregated value we had one minute before. The purpose of this calculation is to get realtime data ingestion rate/topic. Is there a way in kibana to create this kind of visualization? Thank you in advance!
The solution for the ingest rate is using the "Serial difference" aggregation in the classic aggregation types and TSVB, or the "Differences" formula in Lens. And the grouping is available everywhere, just add a second one after the first one.