# Index by query? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 21, 2013, 10:37pm UTC](https://discuss.elastic.co/t/index-by-query/10850 "2013-02-21T22:37:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sbellem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbellem/32/2475_2.png) [@sbellem](https://discuss.elastic.co/u/sbellem)\ **Post date:** [February 21, 2013, 10:37pm UTC](https://discuss.elastic.co/t/index-by-query/10850/1 "2013-02-21T22:37:46Z")
Hi, is it possible (in one step) to index documents that match the results of a given query? That is, similar to the Delete by Query API, can one index by query? Thanks, Sylvain -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 21, 2013, 10:47pm UTC](https://discuss.elastic.co/t/index-by-query/10850/2 "2013-02-21T22:47:20Z")
Do you mean update by query? If so, there is an open issue to support it: > > > \#1583 allows to update individual documents. Update by query will reduce the net…work roundtrips radically if you want to update a number of documents and push work from the client to ES. > > \`\`\` > curl -XPOST localhost:9200/index/type/\_update -d '{ > "query" : { "constant\_score" : { "filter" : { "term" : { "counter" : 0 } } } }, > "script" : "ctx.\_source.counter += count", > "params" : { > "count" : 4 > } > }' > \`\`\` If you really mean index by query, you would need to provide an example, because I don't see how something would be returned by a query unless it already existed. Cheers, Ivan On Thu, Feb 21, 2013 at 2:37 PM, sbellem [sbellem@gmail.com](mailto:sbellem@gmail.com) wrote: > Hi, > > is it possible (in one step) to index documents that match the results of > a given query? That is, similar to the Delete by Query API, can one index > by query? > > Thanks, > Sylvain > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 22, 2013, 10:43am UTC](https://discuss.elastic.co/t/index-by-query/10850/3 "2013-02-22T10:43:05Z")
Hiya > is it possible (in one step) to index documents that match the > results of a given query? That is, similar to the Delete by Query API, > can one index by query? This is something you would need to handle yourself. Depending on which client API you use, you may already have some utilities available to make it easy. For instance, in the Perl API, you could do: ``` $search = $es->scrolled_search( query => { ... whatever query ... }, search_type => 'scan' ); $es->reindex( source => $search, dest_index => 'new_index') ``` If your API doesn't have anything similar, then you can write it yourself. All it consists of is: 1. a scrolled search (with scanning) 2. bulk indexing To implement a scrolled search, run an ordinary search request, with these parameters: - scroll: "1m" this says that you want to take a "snapshot" of the current state of your data, and to keep that around for, eg, "1m" - search\_type: "scan" this search type disables sorting and is very efficient for retrieving large numbers of documents from elasticsearch [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) - size: 1000 the number of docs to return in each request. actually, with 'scan', this means the number of docs to return from EACH shard in each request, eg 5 \* 1000 = max of 5000 docs in each request The above search request will return a scroll ID. You pass that scroll ID to each subsequent "scroll" request /\_search/scroll, until you get no more hits. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. The parameters are: - scroll: "1m" refresh the lock on the scroll snapshot and keep it in place for another one minutes - scroll\_id: "xxxx" the scroll ID returned by the original search request, or by the previous scroll request. You MUST update this scroll ID to have the value of the previous request Each time you call /\_search/scroll you will get another batch of documents. You can reindex those (to a new index, or after making any changes) using the "bulk" API. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![sylvain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain/32/2476_2.png) [@sylvain](https://discuss.elastic.co/u/sylvain)\ **Post date:** [February 22, 2013, 6:03pm UTC](https://discuss.elastic.co/t/index-by-query/10850/4 "2013-02-22T18:03:22Z")
Hi Ivan, yes! I meant update by query, thanks for the link! Best, Sylvain On Thursday, 21 February 2013 23:47:20 UTC+1, Ivan Brusic wrote: > Do you mean update by query? If so, there is an open issue to support it: > [https://github.com/elasticsearch/elasticsearch/issues/1607](https://github.com/elasticsearch/elasticsearch/issues/1607) > > If you really mean index by query, you would need to provide an example, > because I don't see how something would be returned by a query unless it > already existed. > > Cheers, > > Ivan > > On Thu, Feb 21, 2013 at 2:37 PM, sbellem \<[sbe...@gmail.com](mailto:sbe...@gmail.com) \\>wrote: > > > Hi, > > > > is it possible (in one step) to index documents that match the results > > of a given query? That is, similar to the Delete by Query API, can one > > index by query? > > > > Thanks, > > Sylvain > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![sbellem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbellem/32/2475_2.png) [@sbellem](https://discuss.elastic.co/u/sbellem)\ **Post date:** [February 22, 2013, 6:08pm UTC](https://discuss.elastic.co/t/index-by-query/10850/5 "2013-02-22T18:08:18Z")
Hi Ivan, yes! I meant update by query, thanks for the link! Best, Sylvain On Thursday, 21 February 2013 23:47:20 UTC+1, Ivan Brusic wrote: > Do you mean update by query? If so, there is an open issue to support it: > [https://github.com/elasticsearch/elasticsearch/issues/1607](https://github.com/elasticsearch/elasticsearch/issues/1607) > > If you really mean index by query, you would need to provide an example, > because I don't see how something would be returned by a query unless it > already existed. > > Cheers, > > Ivan > > On Thu, Feb 21, 2013 at 2:37 PM, sbellem \<[sbe...@gmail.com](mailto:sbe...@gmail.com) \\>wrote: > > > Hi, > > > > is it possible (in one step) to index documents that match the results > > of a given query? That is, similar to the Delete by Query API, can one > > index by query? > > > > Thanks, > > Sylvain > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![sbellem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbellem/32/2475_2.png) [@sbellem](https://discuss.elastic.co/u/sbellem)\ **Post date:** [February 22, 2013, 6:10pm UTC](https://discuss.elastic.co/t/index-by-query/10850/6 "2013-02-22T18:10:58Z")
Thanks Clint! On Friday, 22 February 2013 11:43:05 UTC+1, Clinton Gormley wrote: > Hiya > > > is it possible (in one step) to index documents that match the > > results of a given query? That is, similar to the Delete by Query API, > > can one index by query? > > This is something you would need to handle yourself. Depending on which > client API you use, you may already have some utilities available to > make it easy. > > For instance, in the Perl API, you could do: > > ``` > $search = $es->scrolled_search( > query => { ... whatever query ... }, > search_type => 'scan' > ); > > $es->reindex( source => $search, dest_index => 'new_index') > > ``` > > If your API doesn't have anything similar, then you can write it > yourself. All it consists of is: > > 1. a scrolled search (with scanning) > 2. bulk indexing > > To implement a scrolled search, run an ordinary search request, with > these parameters: > > - scroll: "1m" > this says that you want to take a "snapshot" of the current state > of your data, and to keep that around for, eg, "1m" > > - search\_type: "scan" > this search type disables sorting and is very efficient for > retrieving large numbers of documents from elasticsearch > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) > > - size: 1000 > the number of docs to return in each request. actually, with > 'scan', this means the number of docs to return from EACH shard > in each request, eg 5 \* 1000 = max of 5000 docs in each request > > The above search request will return a scroll ID. You pass that scroll > ID to each subsequent "scroll" request /\_search/scroll, until you get no > more hits. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/scroll.html) > > The parameters are: > > - scroll: "1m" > refresh the lock on the scroll snapshot and keep it in place for > another one minutes > - scroll\_id: "xxxx" > the scroll ID returned by the original search request, or by the > previous scroll request. You MUST update this scroll ID to have > the value of the previous request > > Each time you call /\_search/scroll you will get another batch of > documents. > > You can reindex those (to a new index, or after making any changes) > using the "bulk" API. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://elasticsearch.org/guide/reference/api/bulk.html) > > clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:50am UTC](https://discuss.elastic.co/t/index-by-query/10850/7 "2017-07-06T02:50:00Z")