Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts

Hello,

I'm running many clusters of ElasticSearch with Kibana, we have version 7.6.2, this is version info:

{
  "name" : "CLUSTER_NAME...",
  "cluster_name" : "CLUSTER_NAME",
  "cluster_uuid" : "4zR...",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "ef4...",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Here are some logs from kibana.log:

{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["reporting","error"],"pid":17962,"message":"ErrorEvent {\n  target:\n   WebSocket {\n     _events:\n      [Object: null prototype] { open: [Function], error: [Function] },\n     _eventsCount: 2,\n     _maxListeners: undefined,\n     readyState: 3,\n     protocol: '',\n     _binaryType: 'nodebuffer',\n     _closeFrameReceived: false,\n     _closeFrameSent: false,\n     _closeMessage: '',\n     _closeTimer: null,\n     _closeCode: 1006,\n     _extensions: {},\n     _receiver: null,\n     _sender: null,\n     _socket: null,\n     _isServer: false,\n     _redirects: 0,\n     url:\n      'ws://127.0.0.1:44141/devtools/browser/58fb58e3-4983-4de0-bd3b-47ad6dd47560',\n     _req: null },\n  type: 'error',\n  message: 'connect ECONNREFUSED 127.0.0.1:44141',\n  error:\n   { Error: connect ECONNREFUSED 127.0.0.1:44141\n       at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)\n     errno: 'ECONNREFUSED',\n     code: 'ECONNREFUSED',\n     syscall: 'connect',\n     address: '127.0.0.1',\n     port: 44141 } }"}
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["reporting","warning"],"pid":17962,"message":"See Chromium's log output at \"/var/lib/kibana/headless_shell-linux/chrome_debug.log\""}

{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["error","plugins","security","authorization"],"pid":17962,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["status","plugin:security@7.6.2","error"],"pid":17962,"state":"red","message":"Status changed from green to red - [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2020-11-30T19:00:50Z","tags":["error","plugins","security","authorization"],"pid":17962,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}

{"type":"log","@timestamp":"2020-11-30T19:00:52Z","tags":["error","plugins","taskManager","taskManager"],"pid":17962,"message":"Failed to poll for work: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting. :: {\"path\":\"/.kibana_task_manager/_update_by_query\",\"query\":{\"ignore_unavailable\":true,\"refresh\":true,\"max_docs\":10,\"conflicts\":\"proceed\"},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.status\\\":\\\"idle\\\"}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"term\\\":{\\\"task.status\\\":\\\"running\\\"}},{\\\"term\\\":{\\\"task.status\\\":\\\"claiming\\\"}}]}},{\\\"range\\\":{\\\"task.retryAt\\\":{\\\"lte\\\":\\\"now\\\"}}}]}}]}},{\\\"bool\\\":{\\\"should\\\":[{\\\"exists\\\":{\\\"field\\\":\\\"task.schedule\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.server-log\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.slack\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.email\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.index\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.pagerduty\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"actions:.webhook\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":1}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"alerting:siem.signals\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"vis_telemetry\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"lens_telemetry\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}}]}}]}}]}},\\\"sort\\\":{\\\"_script\\\":{\\\"type\\\":\\\"number\\\",\\\"order\\\":\\\"asc\\\",\\\"script\\\":{\\\"lang\\\":\\\"painless\\\",\\\"source\\\":\\\"\\\\nif (doc['task.retryAt'].size()!=0) {\\\\n  return doc['task.retryAt'].value.toInstant().toEpochMilli();\\\\n}\\\\nif (doc['task.runAt'].size()!=0) {\\\\n  return doc['task.runAt'].value.toInstant().toEpochMilli();\\\\n}\\\\n    \\\"}}},\\\"seq_no_primary_term\\\":true,\\\"script\\\":{\\\"source\\\":\\\"ctx._source.task.ownerId=params.ownerId; ctx._source.task.status=params.status; ctx._source.task.retryAt=params.retryAt;\\\",\\\"lang\\\":\\\"painless\\\",\\\"params\\\":{\\\"ownerId\\\":\\\"kibana:86dce749-943a-49bb-97ba-e89b81c0a007\\\",\\\"status\\\":\\\"claiming\\\",\\\"retryAt\\\":\\\"2020-11-30T19:01:22.921Z\\\"}}}\",\"statusCode\":500,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"exception\\\",\\\"reason\\\":\\\"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\".kibana_task_manager_1\\\",\\\"node\\\":\\\"1yzPo81-Rl2c2tg1bqhJmg\\\",\\\"reason\\\":{\\\"type\\\":\\\"exception\\\",\\\"reason\\\":\\\"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\\\"}}]},\\\"status\\\":500}\"}"}

I've even tried to increase the number of available scrolls, you can see it by this log line:

{"type":"log","@timestamp":"2020-11-30T19:03:05Z","tags":["error","plugins","security","authorization"],"pid":17962,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500000]. This limit can be set by changing the [search.max_open_scroll_context] setting."}

I have tried to delete .kibana_1 index and .kibana_task_manager_1. But nothing helps.

Also, I'm getting this errors when try to reproduce the error simply by reindex task:

~> curl -sS localhost:9200/_reindex -H "Content-type: application/json" -d '{"source": {"index":".kibana_1"}, "dest":{"index":".backup_kibana_1"}}' | jq .
{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": ".kibana_1",
        "node": "hVa...",
        "reason": {
          "type": "exception",
          "reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
        }
      }
    ]
  },
  "status": 500
}
~> curl -sS localhost:9200/_reindex -H "Content-type: application/json" -d '{"source": {"index":".kibana_task_manager_1"}, "dest":{"index":".backup_kibana_task_manger_1"}}' | jq .
{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": ".kibana_task_manager_1",
        "node": "1yz...",
        "reason": {
          "type": "exception",
          "reason": "Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."
        }
      }
    ]
  },
  "status": 500
}

According to server stats, nobody uses scroll API, only kibana_task_manager is getting higher and higher...

1 Like

By coincidence or not, we started seeing the same issue and error messages yesterday. Version info:

{
  "name" : "NODENAME",
  "cluster_name" : "CLUSTERNAME",
  "cluster_uuid" : "CLUSTERID",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

After a restart of Elasticsearch, the node stats also look completely fine:

"indices" : {
    "search" : {
      "open_contexts" : 0,
      "query_total" : 4703044,
      "query_time_in_millis" : 138223,
      "query_current" : 0,
      "fetch_total" : 930,
      "fetch_time_in_millis" : 2029,
      "fetch_current" : 0,
      "scroll_total" : 437,
      "scroll_time_in_millis" : 3205,
      "scroll_current" : 0,
      "suggest_total" : 0,
      "suggest_time_in_millis" : 0,
      "suggest_current" : 0
    }
}

The issue seems to be caused by the security plugin, which crashes our Kibana instance:

My complaints:

  • There is no way to find any information about current scrolls in the system. No _cat API for that
  • Even after increasing number of scrolls to 500k it still saying that it can't create a new scroll:

Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [500000].

It seems like that these steps can fix it somehow:

  • stop kibana
  • Disable xpack.security in kibana.yaml
  • curl -sS localhost:9200/_search/scroll?scroll_id=_all -XDELETE
  • DELETE ALL KIBANA INDICES !!!!11 (.kibana_1, .kibana_task_manager_1, .security_7, .apm-agent-configuration)
  • start kibana

Please be aware that it will delete everything you have in you kibana, all index patters, all charts, all dashboards

If possible, upgrading Kibana also seems to work. We've upgraded to 7.9.2, without touching configuration and system indices, and do not see the issue anymore.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.