Setting search.allow_expensive_queries to false breaks Kibana Alert Interface

Hello,

I needed to set search.allow_expensive_queries to false because some expensive queries were putting a high load on my cluster and impacting the indexing, so temporarily I made this change.

But after I applied this setting the Kibana Alerts stopped working, the connectors could not be listed if this setting is on.

Looking in Kibana logs, it seems that to list the connectors and send alerts, it performs a joining query, that does not work if search.allow_expensive_queries is set to false.

Is this the intended behaviour? I couldn't find anything about it in the Kibana Alerts documentation, did I miss something or this is undocumented?

This is the log from kibana:

{
  "type": "log",
  "@timestamp": "2021-09-02T15:58:46+00:00",
  "tags": [
    "error",
    "http"
  ],
  "pid": 10998,
  "message": "StatusCodeError: [exception] [joining] queries cannot be executed when 'search.allow_expensive_queries' is set to false.\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4991:19)\n    at IncomingMessage.emit (events.js:327:22)\n    at endReadableNT (internal/streams/readable.js:1327:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21) {\n  status: 500,\n  displayName: 'InternalServerError',\n  path: '/.kibana/_search',\n  query: {},\n  body: {\n    error: {\n      root_cause: [Array],\n      type: 'search_phase_execution_exception',\n      reason: 'all shards failed',\n      phase: 'query',\n      grouped: true,\n      failed_shards: [Array]\n    },\n    status: 500\n  },\n  statusCode: 500,\n  response: `{\"error\":{\"root_cause\":[{\"type\":\"exception\",\"reason\":\"[joining] queries cannot be executed when 'search.allow_expensive_queries' is set to false.\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\".kibana_7.12.1_001\",\"node\":\"8axeqlB0Qr-WRSxCAZxn_g\",\"reason\":{\"type\":\"exception\",\"reason\":\"[joining] queries cannot be executed when 'search.allow_expensive_queries' is set to false.\"}}]},\"status\":500}`,\n  toString: [Function (anonymous)],\n  toJSON: [Function (anonymous)]\n}"
}

Just did a couple more tests.

With search.allow_expensive_queries set to false, the alerts already created will still work, but you won't be able do add new connectors nor see the current connectors, clicking in the Connectors tab in Kibana will giver you the Unable to load Connectors error.

You also won't be able to see the status of active alerts when clicking in the alert, it will show you a 'No items found' message.

This seems to be a bug, I will open a issue in Github.

Thanks for logging the issue Leandro!
Can you please post it here?

Bhavya

Sure, it is this issue.