# SearchContextMissingException during long scroll/scan operations? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 16, 2015, 5:59pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775 "2015-06-16T17:59:01Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![ptrei](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@ptrei](https://discuss.elastic.co/u/ptrei)\ **Post date:** [June 16, 2015, 5:59pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/1 "2015-06-16T17:59:01Z")
Does anyone else have problems with SearchContextMissingExceptions in scroll/scan operations? I have logstash indices which each contain 10s of millions of records. I need to walk over the entire index, processing data from each record. To do this, I'm using the elasticsearch\_py Python library, and Elasticsearch 1.6.0 on a small (4 node) cluster. Here's my code: ``` import elasticsearch import elasticsearch.exceptions import elasticsearch.helpers as helpers es = elasticsearch.Elasticsearch(['http://XXX.XXX.XXX.108:9200'],retry_on_timeout=True) scanResp = helpers.scan(client=es,scroll="5m",index=index_name,timeout="5m",size=1000) resp={} for resp in scanResp: DO STUFF FOR ONE RECORD ``` The processing is handling serveral thousand records a second when it running, so I don't think I'm hitting the 5 minute limit. After an indeterminate amount of time - sometimes quickly sometimes not, I get this stack dump. I've formatted the last part for easier reading, and redacted part of the IP addresses. ``` Traceback (most recent call last): File "/home/ptrei/util/str2int.py", line 190, in mymain() File "/home/ptrei/util/str2int.py", line 177, in mymain process_index(indexname) File "/home/ptrei/util/str2int.py", line 112, in process_index for resp in scanResp: File "/usr/lib/python2.6/site-packages/elasticsearch-1.4.0-py2.6.egg/elasticsearch/helpers/ __init__.py", line 230, in scan resp = client.scroll(scroll_id, scroll=scroll) File "/usr/lib/python2.6/site-packages/elasticsearch-1.4.0-py2.6.egg/elasticsearch/client/utils.py", line 68, in _wrapped return func(*args, params=params, **kwargs) File "/usr/lib/python2.6/site-packages/elasticsearch-1.4.0-py2.6.egg/elasticsearch/client/ __init__.py", line 616, in scroll params=params, body=scroll_id) File "/usr/lib/python2.6/site-packages/elasticsearch-1.4.0-py2.6.egg/elasticsearch/transport.py", line 308, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/lib/python2.6/site-packages/elasticsearch-1.4.0-py2.6.egg/elasticsearch/connection/http_urllib3.py", line 86, in perform_request self._raise_error(response.status, raw_data) File "/usr/lib/python2.6/site-packages/elasticsearch-1.4.0-py2.6.egg/elasticsearch/connection/base.py", line 102, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) elasticsearch.exceptions.NotFoundError: TransportError(404, u'{"_scroll_id":"c2NhbjswOzE7dG90YWxfaGl0czozNzkzNTg5ODs=","took":76,"timed_out":false,"_shards" :{"total":5,"successful":0,"failed":5,"failures":[ {"status":404,"reason":"SearchContextMissingException[No search context found for id [13]]"}, {"status":404,"reason":"RemoteTransportException[[pegasus_101][inet[/XXX.XXX.XXX.101:9300]] [indices:data/read/search[phase/scan/scroll]]]; nested: SearchContextMissingException[No search context found for id [15]]; "}, {"status":404,"reason":"SearchContextMissingException[No search context found for id [14]]"}, {"status":404,"reason":"RemoteTransportException[[pegasus_101][inet[/XXX.XXX.XXX.101:9300]] [indices:data/read/search[phase/scan/scroll]]]; nested: SearchContextMissingException[No search context found for id [14]]; "}, {"status":404,"reason":"SearchContextMissingException[No search context found for id [15]]"}] },"hits":{"total":37935898,"max_score":0.0,"hits":[]}}') ``` My main suspicion is that I'm running this on underpowered hardware (more on the way), but if anyone has other theories or more insight, I'd love to hear it. Searching shows that similar problems have been around for a while. thanks! Peter ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [June 17, 2015, 8:20am UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/2 "2015-06-17T08:20:44Z")
Is your scroll code using the new scroll id which is sent on each scroll response? A common problem here is that people try to use the original scroll id for all scroll requests which would result in errors similar to this. ---
**Author:** ![ptrei](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@ptrei](https://discuss.elastic.co/u/ptrei)\ **Post date:** [June 17, 2015, 3:14pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/3 "2015-06-17T15:14:13Z")
I'm using elasticsearch-py, a python wrapper for the API. I'm under the impression that this handles the scroll-id internally - I certainly don't see it myself. Sometimes, the process runs correctly for many hours before completing, or failing. Failure seems to vaguely correlate with how heavily the ES cluster is being used, which is why I suggest that my HW is to blame. Peter ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [June 19, 2015, 2:36am UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/4 "2015-06-19T02:36:52Z")
could it be the scroll id expired? ---
**Author:** ![ptrei](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@ptrei](https://discuss.elastic.co/u/ptrei)\ **Post date:** [June 26, 2015, 8:19pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/5 "2015-06-26T20:19:21Z")
Jason: The scroll operation sometimes runs for hours without a problem, at others it fails within a few minutes. I don't thinkits an expiration. I set the timeout to 5 minutes, which is way up from the default 10 seconds. I've tried tinkering with the size of the scroll - dropping to 50 (from 800-1000) seems to help, but not entirely. ---
**Author:** ![bmcalindin](https://avatars.discourse-cdn.com/v4/letter/b/ba9def/32.png) [@bmcalindin](https://discuss.elastic.co/u/bmcalindin)\ **Post date:** [April 1, 2016, 11:15am UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/6 "2016-04-01T11:15:56Z")
We've been having the same issue with our cluster. We are doing many scroll queries with the code between queries taking much less time than the scroll timeout. When the query is searching over a large dataset it has been failing with this error after several hundred iterations. Did you ever find a solution here? ---
**Author:** ![Aman\_Astana](https://avatars.discourse-cdn.com/v4/letter/a/839c29/32.png) [@Aman\_Astana](https://discuss.elastic.co/u/Aman_Astana)\ **Post date:** [October 6, 2016, 2:01am UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/7 "2016-10-06T02:01:49Z")
is any solution for this? have anyone tried this on latest version of ES? for me no reason to use ES if this problem can't be solved. ---
**Author:** ![m1cha3lf](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@m1cha3lf](https://discuss.elastic.co/u/m1cha3lf)\ **Post date:** [February 23, 2017, 12:25pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/8 "2017-02-23T12:25:44Z")
we're having exactly the same issue. Did anyone find the exact reason or maybe a workaround? ---
**Author:** ![ptrei](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@ptrei](https://discuss.elastic.co/u/ptrei)\ **Post date:** [February 23, 2017, 6:50pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/9 "2017-02-23T18:50:32Z")
I'm the OP. Sorry guys, I never found an ES solution, and wound up using Splunk to digest my raw data. I have since moved on to other projects. ---
**Author:** ![yuriybash](https://avatars.discourse-cdn.com/v4/letter/y/8491ac/32.png) [@yuriybash](https://discuss.elastic.co/u/yuriybash)\ **Post date:** [June 14, 2017, 8:20pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/10 "2017-06-14T20:20:12Z")
I'm also experiencing this currently, we're on ES 1.7 using the `elasticsearch` python library. ---
**Author:** ![Anusha\_Eagalapati](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@Anusha\_Eagalapati](https://discuss.elastic.co/u/Anusha_Eagalapati)\ **Post date:** [December 18, 2017, 12:58pm UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/11 "2017-12-18T12:58:18Z")
Even i am facing the same issue.Did someone found a solution for this?!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 4:01am UTC](https://discuss.elastic.co/t/searchcontextmissingexception-during-long-scroll-scan-operations/23775/12 "2022-11-04T04:01:27Z")