We are experiencing for a few days now a scenario which 2 nodes are at 100% CPU usage.
I can tell that these are searches as the search pool and search queue are filled, but I cannot figure out from where they are coming from.
I have tried to remove our main API clients, and also stopped Kibana instance, but it solves the problem only for few minutes and then it returns.
CPU Usage 5 days view from Grafana:
CPU Usage 2 days view from Grafana:
Search pool and queue 2 days from Grafana:
Is there a way from tasks, hot_threads or other APIs to see who is the client performing the requests, or what is the searched index pattern of the requests?
I do see index names for regular searches when I use tasks API but for scrolls I cannot see which index is used.
I have attached tasks output for both problematic nodes in Gist:
We are experiencing a serious cluster degradation,