# Atomic Query Dump **URL:** \ **Category:** Elasticsearch\ **Created:** [November 9, 2012, 6:36pm UTC](https://discuss.elastic.co/t/atomic-query-dump/9650 "2012-11-09T18:36:58Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Kevin\_Fox\_2](https://avatars.discourse-cdn.com/v4/letter/k/b5ac83/32.png) [@Kevin\_Fox\_2](https://discuss.elastic.co/u/Kevin_Fox_2)\ **Post date:** [November 9, 2012, 6:36pm UTC](https://discuss.elastic.co/t/atomic-query-dump/9650/1 "2012-11-09T18:36:58Z")
Hi All, A little background first. I have a system where I have a set of rolling indexes. For example Index 22 Index 23, etc. There is an alias set up that points to the most recent, complete index. current -\> 22. After the importer finishes importing the new version of the index, it updates the alias and deletes the older versions. When doing queries against elasticsearch, I work against the current alias and if it 404's, I simply repeat the operation because of an alias delete/create race. This is all working ok for us so far. Now, I want to do one additional thing. I want to be able to take the search queries being issued by the user that are paged, and use those to do a complete, atomic dump of the documents in the query. Basically the same thing as running the query with size=Infinity. I want to do this, so I don't miss or duplicate any entries in the result set. I'm ok if it takes a long time. This does not seem to work though. If I do the search with size=250000000 and search\_type=query\_and\_fetch it doesn't work. It just sits there doing nothing as far as I can tell. Is there a mode similar to this I could use that would? The only other option I can see is the scroll api. I'm not sure that is safe to use with the current alias/delete index rolling that I am doing. Is it safe? If you have an open scroll\_id, what happens when you try and delete the real index? Does the scroll\_id go invalid? Does it try and point to the new data instead of the old resulting in badness? Thanks, Kevin -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [November 10, 2012, 1:22am UTC](https://discuss.elastic.co/t/atomic-query-dump/9650/2 "2012-11-10T01:22:11Z")
If you delete an index the following scroll query will fail with SearchContextMissingException exceptions for each shard. This is as "dangerous" as calling scroll with an expired scrollid. Basically, the scroll query keeps special data structure (SearchContext) representing the current state of search in memory. These SearchContexts are identified by scroll ids. When index is deleted, all SearchContexts are removed as well, and scroll request against a deleted index fails because it cannot find its SearchContexts. In other words its perfectly safe. You just need to add retry logic for SearchContextMissingException that would start the scroll operation all over again. On Friday, November 9, 2012 1:36:58 PM UTC-5, Kevin Fox wrote: > Hi All, > > A little background first. I have a system where I have a set of rolling > indexes. > > For example > Index 22 > Index 23, etc. > > There is an alias set up that points to the most recent, complete index. > > current -\> 22. > > After the importer finishes importing the new version of the index, it > updates the alias and deletes the older versions. > > When doing queries against elasticsearch, I work against the current alias > and if it 404's, I simply repeat the operation because of an alias > delete/create race. > > This is all working ok for us so far. > > Now, I want to do one additional thing. I want to be able to take the > search queries being issued by the user that are paged, and use those to do > a complete, atomic dump of the documents in the query. Basically the same > thing as running the query with size=Infinity. I want to do this, so I > don't miss or duplicate any entries in the result set. I'm ok if it takes a > long time. This does not seem to work though. If I do the search with > size=250000000 and search\_type=query\_and\_fetch it doesn't work. It just > sits there doing nothing as far as I can tell. Is there a mode similar to > this I could use that would? > > The only other option I can see is the scroll api. I'm not sure that is > safe to use with the current alias/delete index rolling that I am doing. Is > it safe? If you have an open scroll\_id, what happens when you try and > delete the real index? Does the scroll\_id go invalid? Does it try and point > to the new data instead of the old resulting in badness? > > Thanks, > Kevin -- ---
**Author:** ![Kevin\_Fox\_2](https://avatars.discourse-cdn.com/v4/letter/k/b5ac83/32.png) [@Kevin\_Fox\_2](https://discuss.elastic.co/u/Kevin_Fox_2)\ **Post date:** [November 13, 2012, 12:24am UTC](https://discuss.elastic.co/t/atomic-query-dump/9650/3 "2012-11-13T00:24:27Z")
Ok. I think I can make it work then. Thank you for your help. Kevin On Friday, November 9, 2012 5:22:11 PM UTC-8, Igor Motov wrote: > If you delete an index the following scroll query will fail > with SearchContextMissingException exceptions for each shard. This is as > "dangerous" as calling scroll with an expired scrollid. Basically, the > scroll query keeps special data structure (SearchContext) representing the > current state of search in memory. These SearchContexts are identified by > scroll ids. When index is deleted, all SearchContexts are removed as well, > and scroll request against a deleted index fails because it cannot find > its SearchContexts. In other words its perfectly safe. You just need to add > retry logic for SearchContextMissingException that would start the scroll > operation all over again. > > On Friday, November 9, 2012 1:36:58 PM UTC-5, Kevin Fox wrote: > > > Hi All, > > > > A little background first. I have a system where I have a set of rolling > > indexes. > > > > For example > > Index 22 > > Index 23, etc. > > > > There is an alias set up that points to the most recent, complete index. > > > > current -\> 22. > > > > After the importer finishes importing the new version of the index, it > > updates the alias and deletes the older versions. > > > > When doing queries against elasticsearch, I work against the current > > alias and if it 404's, I simply repeat the operation because of an alias > > delete/create race. > > > > This is all working ok for us so far. > > > > Now, I want to do one additional thing. I want to be able to take the > > search queries being issued by the user that are paged, and use those to do > > a complete, atomic dump of the documents in the query. Basically the same > > thing as running the query with size=Infinity. I want to do this, so I > > don't miss or duplicate any entries in the result set. I'm ok if it takes a > > long time. This does not seem to work though. If I do the search with > > size=250000000 and search\_type=query\_and\_fetch it doesn't work. It just > > sits there doing nothing as far as I can tell. Is there a mode similar to > > this I could use that would? > > > > The only other option I can see is the scroll api. I'm not sure that is > > safe to use with the current alias/delete index rolling that I am doing. Is > > it safe? If you have an open scroll\_id, what happens when you try and > > delete the real index? Does the scroll\_id go invalid? Does it try and point > > to the new data instead of the old resulting in badness? > > > > Thanks, > > Kevin -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/atomic-query-dump/9650/4 "2017-07-06T03:04:38Z")
Closed on July 6, 2017, 3:04am UTC