# New operations on indices **URL:** \ **Category:** Elasticsearch\ **Created:** [May 16, 2011, 10:12am UTC](https://discuss.elastic.co/t/new-operations-on-indices/4411 "2011-05-16T10:12:09Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\ **Post date:** [May 16, 2011, 10:12am UTC](https://discuss.elastic.co/t/new-operations-on-indices/4411/1 "2011-05-16T10:12:09Z")
Hello. What do You think about adding some new operations on elastic search indices: 1. copying (cloning) an index - making a copy of existing index under different name. Both old and new indices should be separate after cloning and changes to one of them shouldn't be propagated to other (that's difference with aliasing) Currently to achieve this functionality one should: create new index, iterate over types in old index and over documents in index/type and index all these documents to new index. For large indices it's very ineffective operation. 2. Merging indices based on query or type - I'd like to index all my documents from one index matching some query to second index. First operation is very important for me and probably not so hard to implement. And second one would be a nice addition (for example for joining two indices or moving a type from one index to another). ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [May 16, 2011, 11:18am UTC](https://discuss.elastic.co/t/new-operations-on-indices/4411/2 "2011-05-16T11:18:32Z")
I've easily implemented the functionality with the help of the scan search type: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. indexing the full json into the source. Although there is already a ticket for an optimized reindexing functionality: > > > Be able to ask the system to reindex from the saved JSON by document ID or query…. This is useful once we have ES style plugins for manipulating documents that might later change and therefore cause you to want to reindex some set of documents. > \#490 and #491 would let you query by a set of documents indexed before the required change. > > If you are going to store the JSON, you can take advantage of that by reindex requests. > > This might also allow the system to handle schema changes in the future more automatically by reindexing to the new analyzer over time in batch. > > > Probably not a high priority or must have feature, but this would be extremely c…onvenient. > > This would allow one to: > \- Switch gateways > \- Upgrade to new version with incompatible gateway > \- Clone an entire cluster > > Based on this discussion here: > http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/43b534ca360d808f?pli=1 > For large indices it's very ineffective operation. But when not doing the ineffective version you cannot use the functionality for reindexing. Or why would you then need this clone feature? On May 16, 12:12 pm, Wojciech Durczyński [wojciech.durczyn...@comarch.com](mailto:wojciech.durczyn...@comarch.com) wrote: > Hello. > > What do You think about adding some new operations on Elasticsearch > indices: > > 1. copying (cloning) an index - making a copy of existing index under > different name. > Both old and new indices should be separate after cloning and changes > to one of them shouldn't be propagated to other (that's difference > with aliasing) > Currently to achieve this functionality one should: create new index, > iterate over types in old index and over documents in index/type and > index all these documents to new index. For large indices it's very > ineffective operation. > 2. Merging indices based on query or type - I'd like to index all my > documents from one index matching some query to second index. > > First operation is very important for me and probably not so hard to > implement. And second one would be a nice addition (for example for > joining two indices or moving a type from one index to another). ---
**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\ **Post date:** [May 16, 2011, 11:29am UTC](https://discuss.elastic.co/t/new-operations-on-indices/4411/3 "2011-05-16T11:29:52Z")
I'd like to have one index - trunk and some other indices - branches. Branches are initially copies of the trunk but they may be changed separately and may be "committed" to trunk (merged) or deleted. On May 16, 1:18 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote: > I've easily implemented the functionality with the help of the scan > search type: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2011/03/24/new-search-types.html) > > indexing the full json into the source. > > Although there is already a ticket for an optimized reindexing > functionality: > > [Reindex from \_source by document ID or Query · Issue #492 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/492) > > [Add support to reindex into an http endpoint · Issue #514 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/514) > > > For large indices it's very ineffective operation. > > But when not doing the ineffective version you cannot use the > functionality for reindexing. > > Or why would you then need this clone feature? > > On May 16, 12:12 pm, Wojciech Durczyñski > > [wojciech.durczyn...@comarch.com](mailto:wojciech.durczyn...@comarch.com) wrote: > > > Hello. > > > What do You think about adding some new operations on Elasticsearch > > indices: > > > > 1. copying (cloning) an index - making a copy of existing index under > > different name. > > Both old and new indices should be separate after cloning and changes > > to one of them shouldn't be propagated to other (that's difference > > with aliasing) > > Currently to achieve this functionality one should: create new index, > > iterate over types in old index and over documents in index/type and > > index all these documents to new index. For large indices it's very > > ineffective operation. > > 2. Merging indices based on query or type - I'd like to index all my > > documents from one index matching some query to second index. > > > First operation is very important for me and probably not so hard to > > implement. And second one would be a nice addition (for example for > > joining two indices or moving a type from one index to another). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 16, 2011, 7:49pm UTC](https://discuss.elastic.co/t/new-operations-on-indices/4411/4 "2011-05-16T19:49:07Z")
Yes, reindexing the full or part of the index into another index is planned. It gets tricky since some times (or most times, but not in your case) one might want to munge the data before reindexing. Thats why the first priority was to expose the scan search type, so people can implement it themselves. Note, the scan search type supports providing a custom query (and not just match\_all) so you can only scan a subset of the data and index it into another index. On Monday, May 16, 2011 at 2:29 PM, Wojciech Durczyński wrote: > I'd like to have one index - trunk and some other indices - branches. > Branches are initially copies of the trunk but they may be changed > separately and may be "committed" to trunk (merged) or deleted. > > On May 16, 1:18 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote: > > > I've easily implemented the functionality with the help of the scan > > search type: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2011/03/24/new-search-types.html) > > > > indexing the full json into the source. > > > > Although there is already a ticket for an optimized reindexing > > functionality: > > > > [Reindex from \_source by document ID or Query · Issue #492 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/492) > > > > [Add support to reindex into an http endpoint · Issue #514 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/514) > > > > > For large indices it's very ineffective operation. > > > > But when not doing the ineffective version you cannot use the > > functionality for reindexing. > > > > Or why would you then need this clone feature? > > > > On May 16, 12:12 pm, Wojciech Durczyñski > > > > [wojciech.durczyn...@comarch.com](mailto:wojciech.durczyn...@comarch.com) wrote: > > > > > Hello. > > > > > What do You think about adding some new operations on Elasticsearch > > > indices: > > > > > > 1. copying (cloning) an index - making a copy of existing index under > > > different name. > > > Both old and new indices should be separate after cloning and changes > > > to one of them shouldn't be propagated to other (that's difference > > > with aliasing) > > > Currently to achieve this functionality one should: create new index, > > > iterate over types in old index and over documents in index/type and > > > index all these documents to new index. For large indices it's very > > > ineffective operation. > > > 2. Merging indices based on query or type - I'd like to index all my > > > documents from one index matching some query to second index. > > > > > First operation is very important for me and probably not so hard to > > > implement. And second one would be a nice addition (for example for > > > joining two indices or moving a type from one index to another). ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [May 16, 2011, 10:28pm UTC](https://discuss.elastic.co/t/new-operations-on-indices/4411/5 "2011-05-16T22:28:45Z")
> one might want to munge the data before reindexing yeah, I'm actually using scan type + fetching to e.g. clear a certain field of the docs etc (or any 'merge' operation). finally doing bulk indexing to a new copy of the index. Of course this could be tuned to e.g. run without the http overhead (completely lucene side) but it works reasonable fast (~3000 docs/sec) even without any profiling/optimization of my client code. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/new-operations-on-indices/4411/6 "2017-07-06T04:06:01Z")