Throttle Kibana Queries to Elasticsearch

[jeffzhao]

Hi,

In my Elasticsearch+Kibana environment, sometimes, user submitted heavy queries which blow up Elasticsearch cluster after CPU on the data node spiked to 100%.

I know there is room to optimize the query and I know there is no way to throttle Search inside Elasticsearch. I want to know if there is a way to throttle Kibana queries based on certain criteria, such as Elasticsearch cluster CPU usage.

There are two groups of Kibana user: power user and casual user. Ideally I want to set quota of CPU usage for each group.

[lukeelmers]

Hey @jeffzhao,

We don't currently have a mechanism to throttle Kibana requests based on ES cluster stats, however, you might be interested in following some of the work we are doing around async search, which will eventually give users the ability to send a long-running search to the background.

As that feature evolves, we are thinking of moving toward giving admins the ability to view all in-progress async queries, as well as cancel queries that have been running too long, or that are impacting the cluster performance in some way.

There's also the elasticsearch.requestTimeout setting which you can adjust as a failsafe if the CPU spike is also corresponding to slower response times.

If these don't meet your needs, I'd encourage you to submit a feature request with your idea so that we can share it with the team more widely!

Thanks -

Luke

[jeffzhao]

Thanks Luke.

I checked elasticsearch.requestTimeout: 300000 in my environment. But I still have queries running 50+ minutes. Is this possible?

[lukeelmers]

It is possible depending on what version of the stack you are running -- there was an issue fixed in 7.4 where queries disconnected from Kibana were not also being canceled in Elasticsearch.

[jeffzhao]

okay. That makes sense. I'm using a very old version 4.1.