Hello,
I'm running many clusters of ElasticSearch with Kibana, we have version 7.6.2, this is version info:
{
"name" : "CLUSTER_NAME...",
"cluster_name" : "CLUSTER_NAME",
"cluster_uuid" : "4zR...",
"version" : {
"number" : "7.6.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "ef4...",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Here are some logs from kibana.log:
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["reporting","error"],"pid":17962,"message":"ErrorEvent {\n target:\n WebSocket {\n _events:\n [Object: null prototype] { open: [Function], error: [Function] },\n _eventsCount: 2,\n _maxListeners: undefined,\n readyState: 3,\n protocol: '',\n _binaryType: 'nodebuffer',\n _closeFrameReceived: false,\n _closeFrameSent: false,\n _closeMessage: '',\n _closeTimer: null,\n _closeCode: 1006,\n _extensions: {},\n _receiver: null,\n _sender: null,\n _socket: null,\n _isServer: false,\n _redirects: 0,\n url:\n 'ws://127.0.0.1:44141/devtools/browser/58fb58e3-4983-4de0-bd3b-47ad6dd47560',\n _req: null },\n type: 'error',\n message: 'connect ECONNREFUSED 127.0.0.1:44141',\n error:\n { Error: connect ECONNREFUSED 127.0.0.1:44141\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)\n errno: 'ECONNREFUSED',\n code: 'ECONNREFUSED',\n syscall: 'connect',\n address: '127.0.0.1',\n port: 44141 } }"}
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["reporting","warning"],"pid":17962,"message":"See Chromium's log output at \"/var/lib/kibana/headless_shell-linux/chrome_debug.log\""}
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["error","plugins","security","authorization"],"pid":17962,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["status","plugin:security@7.6.2","error"],"pid":17962,"state":"red","message":"Status changed from green to red - [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["error","plugins","security","authorization"],"pid":17962,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}
{"type":"log","@timestamp":"2020-11-30T19:00:52Z","tags":["error","plugins","taskManager","taskManager"],"pid":17962,"message":"Failed to poll for work: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting. :: {\"path\":\"/.kibana_task_manager/_update_by_query\",\"query\":{\"ignore_unavailable\":true,\"refresh\":true,\"max_docs\":10,\"conflicts\":\"proceed\"},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.status\\\":\\\"idle\\\"}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"term\\\":{\\\"task.status\\\":\\\"running\\\"}},{\\\"term\\\":{\\\"task.status\\\":\\\"claiming\\\"}}]}},{\\\"range\\\":{\\\"task.retryAt\\\":{\\\"lte\\\":\\\"now\\\"}}}]}}]}},{\\\"bool\\\":{\\\"should\\\":[{\\\"exists\\\":{\\\"field\\\":\\\"task.schedule\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.server-log\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.slack\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.email\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.index\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.pagerduty\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.webhook\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"alerting:siem.signals\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"vis_telemetry\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"lens_telemetry\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}}]}}]}}]}},\\\"sort\\\":{\\\"_script\\\":{\\\"type\\\":\\\"number\\\",\\\"order\\\":\\\"asc\\\",\\\"script\\\":{\\\"lang\\\":\\\"painless\\\",\\\"source\\\":\\\"\\\\nif (doc['task.retryAt'].size()!=0) {\\\\n return doc['task.retryAt'].value.toInstant().toEpochMilli();\\\\n}\\\\nif (doc['task.runAt'].size()!=0) {\\\\n return doc['task.runAt'].value.toInstant().toEpochMilli();\\\\n}\\\\n \\\"}}},\\\"seq_no_primary_term\\\":true,\\\"script\\\":{\\\"source\\\":\\\"ctx._source.task.ownerId=params.ownerId; ctx._source.task.status=params.status; ctx._source.task.retryAt=params.retryAt;\\\",\\\"lang\\\":\\\"painless\\\",\\\"params\\\":{\\\"ownerId\\\":\\\"kibana:86dce749-943a-49bb-97ba-e89b81c0a007\\\",\\\"status\\\":\\\"claiming\\\",\\\"retryAt\\\":\\\"2020-11-30T19:01:22.921Z\\\"}}}\",\"statusCode\":500,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"exception\\\",\\\"reason\\\":\\\"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\".kibana_task_manager_1\\\",\\\"node\\\":\\\"1yzPo81-Rl2c2tg1bqhJmg\\\",\\\"reason\\\":{\\\"type\\\":\\\"exception\\\",\\\"reason\\\":\\\"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\\\"}}]},\\\"status\\\":500}\"}"}
I've even tried to increase the number of available scrolls, you can see it by this log line:
{"type":"log","@timestamp":"2020-11-30T19:03:05Z","tags":["error","plugins","security","authorization"],"pid":17962,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500000]. This limit can be set by changing the [search.max_open_scroll_context] setting."}
I have tried to delete .kibana_1 index and .kibana_task_manager_1. But nothing helps.
Also, I'm getting this errors when try to reproduce the error simply by reindex task:
~> curl -sS localhost:9200/_reindex -H "Content-type: application/json" -d '{"source": {"index":".kibana_1"}, "dest":{"index":".backup_kibana_1"}}' | jq .
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": ".kibana_1",
"node": "hVa...",
"reason": {
"type": "exception",
"reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
}
}
]
},
"status": 500
}
~> curl -sS localhost:9200/_reindex -H "Content-type: application/json" -d '{"source": {"index":".kibana_task_manager_1"}, "dest":{"index":".backup_kibana_task_manger_1"}}' | jq .
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": ".kibana_task_manager_1",
"node": "1yz...",
"reason": {
"type": "exception",
"reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
}
}
]
},
"status": 500
}
According to server stats, nobody uses scroll API, only kibana_task_manager is getting higher and higher...
