How to handle gaps in date histogram (missing values) in Kibana and Elastic Search?

[smlbiobot (SML)]

Our indices have missing values from time to time, and when that happens, what I’d like to see in date histogram plots is to simply ignore missing values and connect the lines before and after. Please see this example plot:

The way that I’ve been getting around this is to increase the time interval:

But is there in fact a way native in Kibana to handle this?

Although this question is specific to Kibana, I am also interested in knowing if there are any internal “smoothing / auto-averaging” flags when constructing the aggregations in Elastic Search. We usually use Kibana to pre-visualize what we want to output, but eventually write a program to do the visualizations.

At the moment, we work around this by processing the data after we have gotten the aggregated results. But if there is in fact a way for ES to handle it natively it would save a lot of post-processing time.

Thanks!

[wylie (Wylie Conlon)]

Your workaround of setting the time interval is the same as what I would recommend. The only alternative is to use a Lens line chart in 7.9 or higher, where we support fitting functions: https://github.com/elastic/kibana/pull/69820

We are planning on adding more support for fitting on area charts in a future version.

[smlbiobot (SML)]

Besides Kibana. I would also like to see some type of auto interpolation from Elasticsearch also. Some type of flag to enable it would be great.

I understand that empty data is best retuned as none or empty, but if the caller (we) ask for the data to be interpolated then I think that it is a valid use case.

It would save us the extra step of post processing these gaps after the response is returned.

Thanks!

[wylie (Wylie Conlon)]

Unfortunately that isn't a feature in Elasticsearch. We have previously asked for this, but they pushed back on this feature idea. You can see the reasoning here: https://github.com/elastic/elasticsearch/issues/24747

[smlbiobot (SML)]

I see. I hate to say it but his reasoning for not implementing it is well-argued.

I don’t have a better counter-argument than the people who had chimed in and argued for the case.

I guess I will look into your suggestions for the alternatives. Sadly we are slightly behind here as we have yet to convert all of our data that‘s compatible with 7 and we are in fact at 6.8.13 at the moment.

Thanks for your help!