Since the beginning, Kibana has always been about taking huge volumes of data in Elasticsearch and making sense of it visually. Over time, Kibana has stretched the limits of Elasticsearch, and we’ve needed to make changes in Elasticsearch in order to more powerfully enable users of Kibana.
The Problem
If you’re a Kibana user, you might not realize everything that goes into visualizing huge volumes of data. Elasticsearch is able to efficiently query millions (or even billions) of records and give you results quickly. There are times, however, that a dashboard which includes many visualizations might execute intense queries to Elasticsearch that can take seconds (or even minutes) to run.
As a Kibana user, if a query is taking long to execute, there are a few things that might run through your mind:
- Is something wrong? Maybe I should resubmit this query.
- This query is taking too long. I’m going somewhere else in Kibana.
- Something in this query isn’t quite right. I want to change it and resubmit.
- Maybe I’ll grab a cup of coffee and wait. (cue Jeopardy music)
In the first three situations above, extra work was happening. The initial query continued to run, even though the user no longer cared about its results (taking up valuable system resources), even if a user submitted a completely new query or navigated away from the page before completion.
So, what could we do to fix this?
The Solution
How could we optimize Kibana so that valuable system resources weren't used on queries that had been discarded? Elasticsearch has Task Management APIs that allow cancelling tasks, but the vast majority of Kibana users don’t have privileges to access those APIs.
The Kibana and Elasticsearch teams got together and discussed a new approach. What if, when a connection to Elasticsearch was closed, Elasticsearch intelligently cleaned up the tasks associated with that query? And what if, when a Kibana user resubmitted a query, navigated away, or changed the query, Kibana intelligently closed the connection to Elasticsearch? It was a win-win approach that avoided the aforementioned security implications.
We’re proud to announce that all of this has come together in 7.5. (Administrators of large Elasticsearch clusters, feel free to breathe a sigh of relief!)
Caveats
The majority of visualizations support smart query cancellation, but there are still a few places that we still need to add support. You can follow this meta issue on Github for updates.