Trying to create too many scroll contexts for .security index

[munkyu (munkyu im)]

Hello,

I'm using elastic stack 7.6.2
My Kibana failed to log in with superuser which was done until yesterday.
I checked kibana and elasticsearch verbose log.

Here's the log messages

in kibana

{"type":"log","@timestamp":"2020-07-30T03:45:28Z","tags":["debug","elasticsearch","security","query"],"pid":31854,"message":"500\nGET /_security/privilege/kibana-.kibana\n"}
{"type":"log","@timestamp":"2020-07-30T03:45:28Z","tags":["error","plugins","security","authorization"],"pid":31854,"message":"Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: [exception] Trying to create too many scroll contexts. Must be less than or equal to: [1024]. This limit can be set by changing the [search.max_open_scroll_context] setting."}

in elasticsearch

...
[2020-07-30T17:38:57,630][DEBUG][o.e.a.s.TransportSearchAction] [es-1] [.security-7][0], node[GrhFbKisRFWXvgiyHMAlrg], [P], s[STARTED], a[id=sArEjZbMSkC2MGaAzkIvFw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.security], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=5m}, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":1000,"query":{"bool":{"filter":[{"term":{"type":{"value":"application-privilege","boost":1.0}}},{"terms":{"application":["kibana-.kibana"],"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":[],"excludes":[]}}}]
org.elasticsearch.transport.RemoteTransportException: [es-7][xx.xxx.xx.xxx:10350][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [1024]. This limit can be set by changing the [search.max_open_scroll_context] setting.
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:549) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:343) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-07-30T17:38:57,631][DEBUG][o.e.a.s.TransportSearchAction] [es-1] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [1024]. This limit can be set by changing the [search.max_open_scroll_context] setting.

So, the issue came from API: _security/privilege/kibana-.kibana which is calling the ES API below and seems to be failed to scoll context. (without scroll option, works well)

POST /.security/_search?scroll=5m -d '{"size":1000,"query":{"bool":{"filter":[{"term":{"type":{"value":"application-privilege","boost":1}}},{"terms":{"application":["kibana-.kibana"],"boost":1}}],"adjust_pure_negative":true,"boost":1}},"_source":{"includes":[],"excludes":[]}} 

This is very wired, because current scroll value is almost zero when I check from node stat API

      "indices" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 3185350,
          "query_time_in_millis" : 61877211,
          "query_current" : 0,
          "fetch_total" : 155992,
          "fetch_time_in_millis" : 12592045,
          "fetch_current" : 0,
          "scroll_total" : 17525,
          "scroll_time_in_millis" : 2007247412,
          "scroll_current" : 0,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        }
      }
    }

and scroll API on other indices work well.
Even I set "search.max_open_scroll_context" to huge value, the same error was reproduced.

I think security and authentication process related to kibana has error on my elastic stack.
Other indices work well with security.
I've never seen this problem before.
How can I handle this? I don't use kibana dashboard or index pattern, only use for stack monitoring. So it's OK to remove and restart all kibana related indices. If I delete .security index is it work?
or Any good idea to solve this problem?

Anyone, please guide.

Thanks,
Munkyu