Need to reproduce X-Pack Monitoring's Indexing Rate graph in a Kibana Timelion graph - how?

[Neil_M (Neil Movold)]

I have a need to reproduce both the Search Rate and Indexing Rate graphs from the X-Pack Monitoring application in a timelion graph so that it can be put into a dashboard that can be embedded in an external web page.

I have looked at the fields in the .monitoring index, but cannot figure out how the graphs were built.

Is it possible to reproduce these graphs?

I am currently running Elasticsearch Stack Version 5.3. We are transitioning to the latest ELK 7.x in early 2020, but I need to get this done now with 5.3.

Thanks for your guidance.

Cheers,
Neil

[chrisronline (Chris Roberson)]

Hi @Neil_M,

Welcome and great questions!

I'd be happy to show you the code path to figuring out these things, but I'd also like to warn that building manual charts for system indices (which start with .) are not officially supported, as we may change the mappings without notice in any release.

Search Rate

Since this in on the Elasticsearch overview page, we can see those metrics listed here. Then, that is mapped to this metric here, which shows the field name is: indices_stats._all.total.search.query_total. Once you know that, you can double check the class definition for the metric which will help understand any post processing we might do with the data. For example, a derivative: true means the value is a counter and we use a derivative aggregation.

Indexing Rate

Start here then look here and you'll see the field is: indices_stats._all.total.indexing.index_total.

Hope this helps and good luck!

[Neil_M (Neil Movold)]

Thanks @chrisronline for your guidance! I'll dig into that and see where it leads me. I fully understand your point about the risk of using system indices.

[Neil_M (Neil Movold)]

Chris, it worked perfectly! Took a bit to figure out the right timelion functions to use, but I fully replicated the graph. I was also able to overlay the previous day's data, which is quite helpful. Thanks again for your guidance!