Elasticsearch 7.0.1 - Trying to create too many scroll contexts. Must be less than or equal to: [500]

[cyb3rward0g (Roberto Rodriguez)]

Good afternoon,

I upgraded my stack to 7.0.1 and I am using Elastalert master 05172019. When I run Elasticsearch and elastalert, it alls seems to be working fine. However, the moment I ingest data, and the rules start hitting the indices, I get the following messages:

ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.016s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.007s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-sysmon-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.007s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.005s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.010s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')

I updated the max_open_scroll_context to 5000 and I still get the same errors.

curl -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d'{
    "persistent" : {
        "search.max_open_scroll_context": 5000
    },
    "transient": {
        "search.max_open_scroll_context": 5000
    }
}
'

I opened an issue in Github for Elastalert and I have all my latest notes in there. We are not sure if this is an Elasticsearch issue. Thank you in advance for all the help! https://github.com/Yelp/elastalert/issues/2249

[cyb3rward0g (Roberto Rodriguez)]

These are some of the error logs I have from Elasticsearch:

{"type": "server", "timestamp": "2019-05-18T17:31:47,369+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "helk-cluster", "node.name": "helk-1", "cluster.uuid": "fAUmGpuTTZqA7Gxnzs4jGQ", "node.id": "Y9alb0wgTwO1Dj_8yEJwYw",  "message": "path: /logs-endpoint-winevent-security-*/_search, params: {size=5000, ignore_unavailable=true, scroll=1m, index=logs-endpoint-winevent-security-*, _source_includes=@timestamp,*}" , 
"stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1236) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1210) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:366) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:360) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:354) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1069) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:835) [?:?]",
"Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.",

[cyb3rward0g (Roberto Rodriguez)]

These are some of the commands sent from Elastalert:

curl -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/elastalert_status_error/_doc?pretty' -d '{
  "@timestamp": "2019-05-18T17:35:30.308948Z",
  "data": {
    "query": {
      "query": {
        "bool": {
          "filter": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gt": "2019-05-18T17:33:30.298127Z",
                      "lte": "2019-05-18T17:35:30.298127Z"
                    }
                  }
                },
                {
                  "query_string": {
                    "query": "(event_id:(\"675\" \"4768\" \"4769\" \"4771\") AND ticket_failure_code:(\"0x9\" \"0xA\" \"0xB\" \"0xF\" \"0x10\" \"0x11\" \"0x13\" \"0x14\" \"0x1A\" \"0x1F\" \"0x21\" \"0x22\" \"0x23\" \"0x24\" \"0x26\" \"0x27\" \"0x28\" \"0x29\" \"0x2C\" \"0x2D\" \"0x2E\" \"0x2F\" \"0x31\" \"0x32\" \"0x3E\" \"0x3F\" \"0x40\" \"0x41\" \"0x43\" \"0x44\"))"
                  }
                }
              ]
            }
          }
        }
      },
      "sort": [
        {
          "@timestamp": {
            "order": "asc"
          }
        }
      ]
    },
    "rule": "Kerberos-Manipulation_0"
  },
  "message": "Error running query: TransportError(500, u\u0027search_phase_execution_exception\u0027, u\u0027Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\u0027)",
  "traceback": [
    "Traceback (most recent call last):",
    "  File \"/usr/local/lib/python2.7/dist-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py\", line 352, in get_hits",
    "    **extra_args",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py\", line 84, in _wrapped",
    "    return func(*args, params=params, **kwargs)",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py\", line 818, in search",
    "    \"GET\", _make_path(index, \"_search\"), params=params, body=body",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 353, in perform_request",
    "    timeout=timeout,",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py\", line 143, in perform_request",
    "    self._raise_error(response.status_code, raw_data)",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py\", line 168, in _raise_error",
    "    status_code, error_message, additional_info",
    "TransportError: TransportError(500, u\u0027search_phase_execution_exception\u0027, u\u0027Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\u0027)"
  ]
}'