Post transformation from Elasticsearch to display in Kibana


It is more a discussion rather that a direct question.

Imagine you have the following use case:

You insert the type of document as show bellow from different sources:

  "count": 10000,
  "offset": 1234,

The count field is more like a wall clock, it goes towards incrementing but can decrement.
The offset field in contrast is more like a monotonic clock, it always increments.

What I need is only the last count value from the document that has the highest offset value and ideally in Kibana.

So I have used the equivalent of max(count) to display the count. But obviously since count can go backward, this solution is not working properly.

I then used in kibana, the max(offset) and then terms aggregation of size 1 sorted by max(offset). Which gives me the right value of course.

The approach works but Kibana displays max(offset) really big, and the result of the aggregation in tiny -- which I ideally either want the opposite, or just not display max(offset) at all.

I am potentially stuck when using Kibana if I don't remodel data from elasticsearch. My idea is to have workers that periodically query elasticsearch then do some modeling -- possibly doing some other query -- to then re-inject the data in elasticsearc.
This data will be optimized to be displayed in Kibana.

I am wondering how people do it in general, if they used already installed tools and tricks that I don't know about.. Do people remodel datas to be displayed in kibana independently? And if so, is it custom made?

Thank you!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.