# Reindexing with new mapping **URL:** \ **Category:** Elasticsearch\ **Created:** [September 12, 2011, 12:00am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348 "2011-09-12T00:00:30Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [September 12, 2011, 12:00am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/1 "2011-09-12T00:00:30Z")
We are in a situation where we need to reindex a few hundred million docs (add some indexed fields, add some new fields). We hope to do this online by changing the mapping then performing updates on all the old docs to reindex them with the new mapping. Along these lines, I have a couple of questions: 1. Can a field mapping be changed from "index":"no" to to indexed using the put mapping API? 2. Can the default analyzer be changed with put mapping API? thanks, Curtis ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 12, 2011, 12:15am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/2 "2011-09-12T00:15:52Z")
Hi Curtis, In case if it's not possible to update mapping as you describe (which is my opinon), I asked for a feature request to allow recreating data from an ES instance to another one with a river. > > > As discussed in the mailing list : http://elasticsearch-users.115913.n3.nabble.c…om/How-to-reindex-an-ES-index-tp3089964p3089964.html > > It would be nice to be able to reindex data from an ES instance using the \`\_source\` field of previously stored documents. > > With it, we could : > \- Modify the mapping and ask for reindexing (even in the same cluster) documents stored in oldindex to a newindex index. The new mapping will be defined in newindex. > \- Migrate easily from an ES version to another if needed > \- Do many cool things that I can't imagine right now ;-) > > Thanks Not sure I have enough ES background to start writing it... ☹ David 😉 Le 12 sept. 2011 à 02:00, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) a écrit : > We are in a situation where we need to reindex a few hundred million docs (add some indexed fields, add some new fields). We hope to do this online by changing the mapping then performing updates on all the old docs to reindex them with the new mapping. > > Along these lines, I have a couple of questions: > > 1. Can a field mapping be changed from "index":"no" to to indexed using the put mapping API? > 2. Can the default analyzer be changed with put mapping API? > > thanks, > > Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 12, 2011, 9:19am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/3 "2011-09-12T09:19:05Z")
You can't change a field from not being indexed to being indexed. You can change the default analyzer, but this will only affect future documents indexed (by closing the index, updating the index settings, and then opening it). If you end up reindexing the data, why not just index it into a new index with the new mappings? On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > We are in a situation where we need to reindex a few hundred million docs > (add some indexed fields, add some new fields). We hope to do this online > by changing the mapping then performing updates on all the old docs to > reindex them with the new mapping. > > Along these lines, I have a couple of questions: > > 1. Can a field mapping be changed from "index":"no" to to indexed using > the put mapping API? > 2. Can the default analyzer be changed with put mapping API? > > thanks, > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [September 12, 2011, 6:18pm UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/4 "2011-09-12T18:18:52Z")
Ok, thanks. In that case I will go the route of indexing into a new index. Curtis On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > You can't change a field from not being indexed to being indexed. You can > change the default analyzer, but this will only affect future documents > indexed (by closing the index, updating the index settings, and then opening > it). > > If you end up reindexing the data, why not just index it into a new index > with the new mappings? > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > We are in a situation where we need to reindex a few hundred million docs > > (add some indexed fields, add some new fields). We hope to do this online > > by changing the mapping then performing updates on all the old docs to > > reindex them with the new mapping. > > > > Along these lines, I have a couple of questions: > > > > 1. Can a field mapping be changed from "index":"no" to to indexed using > > the put mapping API? > > 2. Can the default analyzer be changed with put mapping API? > > > > thanks, > > > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [September 13, 2011, 6:05pm UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/5 "2011-09-13T18:05:01Z")
Ok, I'm going with the strategy of creating a new index, but I want to do the reindexing all online using aliases. That leads to a couple of alias questions: 1. How are doc ids treated when you do a get (or search) operation on an alias with multiple underlying indices? 2. What happens if two docs with the same id exist in two of the underlying indices? Is there some precedence or order to which doc is returned? 3. Does the index status API work with aliases? For example, can I wait for yellow status on an alias rather than listing all the underlying indices? Thanks again, Curtis On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > Ok, thanks. In that case I will go the route of indexing into a new index. > > Curtis > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > You can't change a field from not being indexed to being indexed. You can > > change the default analyzer, but this will only affect future documents > > indexed (by closing the index, updating the index settings, and then opening > > it). > > > > If you end up reindexing the data, why not just index it into a new index > > with the new mappings? > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > We are in a situation where we need to reindex a few hundred million docs > > > (add some indexed fields, add some new fields). We hope to do this online > > > by changing the mapping then performing updates on all the old docs to > > > reindex them with the new mapping. > > > > > > Along these lines, I have a couple of questions: > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed using > > > the put mapping API? > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > thanks, > > > > > > Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 13, 2011, 8:16pm UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/6 "2011-09-13T20:16:41Z")
I think you are trying to use the aliases wrongly. When you reindex and want to do "hot" replace of indices, you use a single alias pointing to a single index as the one the "client" uses. For example, have alias1 point to index1. Then, you reindex the data into index2, and once its done, you switch (in the same command) alias1 to point to index2 from index1. In the above usecase, there is no point where an alias is pointing to more than one index. Of course, you can have an alias point to more than one index, but it only really make sense when searching, and in this case, its the same as searching across several indices, where the index is part of the "uniqueness" of the document. On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > Ok, I'm going with the strategy of creating a new index, but I want to do > the reindexing all online using aliases. > > That leads to a couple of alias questions: > > 1. How are doc ids treated when you do a get (or search) operation on an > alias with multiple underlying indices? > 2. What happens if two docs with the same id exist in two of the > underlying indices? Is there some precedence or order to which doc is > returned? > 3. Does the index status API work with aliases? For example, can I wait > for yellow status on an alias rather than listing all the underlying > indices? > > Thanks again, > > Curtis > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > Ok, thanks. In that case I will go the route of indexing into a new > > index. > > > > Curtis > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > You can't change a field from not being indexed to being indexed. You can > > > change the default analyzer, but this will only affect future documents > > > indexed (by closing the index, updating the index settings, and then opening > > > it). > > > > > > If you end up reindexing the data, why not just index it into a new index > > > with the new mappings? > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > We are in a situation where we need to reindex a few hundred million > > > > docs (add some indexed fields, add some new fields). We hope to do this > > > > online by changing the mapping then performing updates on all the old docs > > > > to reindex them with the new mapping. > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed using > > > > the put mapping API? > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > thanks, > > > > > > > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [September 13, 2011, 9:02pm UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/7 "2011-09-13T21:02:11Z")
Ok, I think that answers my questions. The reason I proposed using multiple indices with aliases is that I want to be able to take in new data while the migration is taking place. For example: index\_old: index with old mapping and data index\_new: index with new mapping At some point, switch writes to index\_new, so it starts filling up with new data. Then, migrate data from old to new while the system is online and still taking new data. I want to be able to search across both old and new during this process, similar to an online rebuild of a database index. Is there a better way I could be doing this? Curtis On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > I think you are trying to use the aliases wrongly. When you reindex and > want to do "hot" replace of indices, you use a single alias pointing to a > single index as the one the "client" uses. For example, have alias1 point to > index1. Then, you reindex the data into index2, and once its done, you > switch (in the same command) alias1 to point to index2 from index1. > > In the above usecase, there is no point where an alias is pointing to more > than one index. Of course, you can have an alias point to more than one > index, but it only really make sense when searching, and in this case, its > the same as searching across several indices, where the index is part of the > "uniqueness" of the document. > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > Ok, I'm going with the strategy of creating a new index, but I want to do > > the reindexing all online using aliases. > > > > That leads to a couple of alias questions: > > > > 1. How are doc ids treated when you do a get (or search) operation on an > > alias with multiple underlying indices? > > 2. What happens if two docs with the same id exist in two of the > > underlying indices? Is there some precedence or order to which doc is > > returned? > > 3. Does the index status API work with aliases? For example, can I wait > > for yellow status on an alias rather than listing all the underlying > > indices? > > > > Thanks again, > > > > Curtis > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Ok, thanks. In that case I will go the route of indexing into a new > > > index. > > > > > > Curtis > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > You can't change a field from not being indexed to being indexed. You > > > > can change the default analyzer, but this will only affect future documents > > > > indexed (by closing the index, updating the index settings, and then opening > > > > it). > > > > > > > > If you end up reindexing the data, why not just index it into a new > > > > index with the new mappings? > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > We are in a situation where we need to reindex a few hundred million > > > > > docs (add some indexed fields, add some new fields). We hope to do this > > > > > online by changing the mapping then performing updates on all the old docs > > > > > to reindex them with the new mapping. > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed > > > > > using the put mapping API? > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > thanks, > > > > > > > > > > Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 13, 2011, 9:15pm UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/8 "2011-09-13T21:15:37Z")
Taking data while having the old index around can be problematic if you have updates / deletes. You might need to still apply the changes to the old index, and buffer them while you reindex to apply them again or something like that. On Wed, Sep 14, 2011 at 12:02 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > Ok, I think that answers my questions. > > The reason I proposed using multiple indices with aliases is that I want to > be able to take in new data while the migration is taking place. For > example: > > index\_old: index with old mapping and data > index\_new: index with new mapping > > At some point, switch writes to index\_new, so it starts filling up with new > data. > Then, migrate data from old to new while the system is online and still > taking new data. > > I want to be able to search across both old and new during this process, > similar to an online rebuild of a database index. > > Is there a better way I could be doing this? > > Curtis > > On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > I think you are trying to use the aliases wrongly. When you reindex and > > want to do "hot" replace of indices, you use a single alias pointing to a > > single index as the one the "client" uses. For example, have alias1 point to > > index1. Then, you reindex the data into index2, and once its done, you > > switch (in the same command) alias1 to point to index2 from index1. > > > > In the above usecase, there is no point where an alias is pointing to more > > than one index. Of course, you can have an alias point to more than one > > index, but it only really make sense when searching, and in this case, its > > the same as searching across several indices, where the index is part of the > > "uniqueness" of the document. > > > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Ok, I'm going with the strategy of creating a new index, but I want to do > > > the reindexing all online using aliases. > > > > > > That leads to a couple of alias questions: > > > > > > 1. How are doc ids treated when you do a get (or search) operation on an > > > alias with multiple underlying indices? > > > 2. What happens if two docs with the same id exist in two of the > > > underlying indices? Is there some precedence or order to which doc is > > > returned? > > > 3. Does the index status API work with aliases? For example, can I wait > > > for yellow status on an alias rather than listing all the underlying > > > indices? > > > > > > Thanks again, > > > > > > Curtis > > > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > Ok, thanks. In that case I will go the route of indexing into a new > > > > index. > > > > > > > > Curtis > > > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > You can't change a field from not being indexed to being indexed. You > > > > > can change the default analyzer, but this will only affect future documents > > > > > indexed (by closing the index, updating the index settings, and then opening > > > > > it). > > > > > > > > > > If you end up reindexing the data, why not just index it into a new > > > > > index with the new mappings? > > > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > > > We are in a situation where we need to reindex a few hundred million > > > > > > docs (add some indexed fields, add some new fields). We hope to do this > > > > > > online by changing the mapping then performing updates on all the old docs > > > > > > to reindex them with the new mapping. > > > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed > > > > > > using the put mapping API? > > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > > > thanks, > > > > > > > > > > > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [September 13, 2011, 9:23pm UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/9 "2011-09-13T21:23:04Z")
Good point, I'll have to think about this some more. Curtis On Tue, Sep 13, 2011 at 2:15 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Taking data while having the old index around can be problematic if you > have updates / deletes. You might need to still apply the changes to the old > index, and buffer them while you reindex to apply them again or something > like that. > > On Wed, Sep 14, 2011 at 12:02 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > Ok, I think that answers my questions. > > > > The reason I proposed using multiple indices with aliases is that I want > > to be able to take in new data while the migration is taking place. For > > example: > > > > index\_old: index with old mapping and data > > index\_new: index with new mapping > > > > At some point, switch writes to index\_new, so it starts filling up with > > new data. > > Then, migrate data from old to new while the system is online and still > > taking new data. > > > > I want to be able to search across both old and new during this process, > > similar to an online rebuild of a database index. > > > > Is there a better way I could be doing this? > > > > Curtis > > > > On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > I think you are trying to use the aliases wrongly. When you reindex and > > > want to do "hot" replace of indices, you use a single alias pointing to a > > > single index as the one the "client" uses. For example, have alias1 point to > > > index1. Then, you reindex the data into index2, and once its done, you > > > switch (in the same command) alias1 to point to index2 from index1. > > > > > > In the above usecase, there is no point where an alias is pointing to > > > more than one index. Of course, you can have an alias point to more than one > > > index, but it only really make sense when searching, and in this case, its > > > the same as searching across several indices, where the index is part of the > > > "uniqueness" of the document. > > > > > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > Ok, I'm going with the strategy of creating a new index, but I want to > > > > do the reindexing all online using aliases. > > > > > > > > That leads to a couple of alias questions: > > > > > > > > 1. How are doc ids treated when you do a get (or search) operation on > > > > an alias with multiple underlying indices? > > > > 2. What happens if two docs with the same id exist in two of the > > > > underlying indices? Is there some precedence or order to which doc is > > > > returned? > > > > 3. Does the index status API work with aliases? For example, can I > > > > wait for yellow status on an alias rather than listing all the underlying > > > > indices? > > > > > > > > Thanks again, > > > > > > > > Curtis > > > > > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > Ok, thanks. In that case I will go the route of indexing into a new > > > > > index. > > > > > > > > > > Curtis > > > > > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > > > You can't change a field from not being indexed to being indexed. You > > > > > > can change the default analyzer, but this will only affect future documents > > > > > > indexed (by closing the index, updating the index settings, and then opening > > > > > > it). > > > > > > > > > > > > If you end up reindexing the data, why not just index it into a new > > > > > > index with the new mappings? > > > > > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > > > > > We are in a situation where we need to reindex a few hundred million > > > > > > > docs (add some indexed fields, add some new fields). We hope to do this > > > > > > > online by changing the mapping then performing updates on all the old docs > > > > > > > to reindex them with the new mapping. > > > > > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed > > > > > > > using the put mapping API? > > > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > Curtis ---
**Author:** ![Stephen\_Beeson](https://avatars.discourse-cdn.com/v4/letter/s/58f4c7/32.png) [@Stephen\_Beeson](https://discuss.elastic.co/u/Stephen_Beeson)\ **Post date:** [July 24, 2012, 7:07am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/10 "2012-07-24T07:07:38Z")
Hi Guys, I found this thread was a useful starting point to handle my own simliar issues. I did come up with another solution but wanted to run it by you to see if you can spot any issues with it. The basic concept is to perform all reads of an index with one alias and all writes with another, eg. index\_read and index\_write When a reindex was performed, another index would be created and added to the index\_write alias ``` index_read -> old_index index_write -> old_index new_index ``` This way all updates come through on the new index while it is being populated and the old index continues to return newly updated documents. Once the reindexing is complete, I'd remove the old index and add the new to the read alias, and remove the old index from the write alias. Then I could delete the old and it should all be quite seemless. I have my own python code to access elasticsearch so the code that references the two aliases in all in the one place. Does this seems like a reasonable approach? I think this could be made even easier if there was a mechanism similar to the "search\_routing" and "index\_routing" features. Perhaps the aliases could have add\_search and add\_index actions. Kimchy, does this sound like a sensible feature? Thanks Stephen On Wednesday, September 14, 2011 7:23:04 AM UTC+10, Curtis Caravone wrote: > Good point, I'll have to think about this some more. > > Curtis > > On Tue, Sep 13, 2011 at 2:15 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Taking data while having the old index around can be problematic if you > > have updates / deletes. You might need to still apply the changes to the > > old index, and buffer them while you reindex to apply them again or > > something like that. > > > > On Wed, Sep 14, 2011 at 12:02 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Ok, I think that answers my questions. > > > > > > The reason I proposed using multiple indices with aliases is that I want > > > to be able to take in new data while the migration is taking place. For > > > example: > > > > > > index\_old: index with old mapping and data > > > index\_new: index with new mapping > > > > > > At some point, switch writes to index\_new, so it starts filling up with > > > new data. > > > Then, migrate data from old to new while the system is online and still > > > taking new data. > > > > > > I want to be able to search across both old and new during this process, > > > similar to an online rebuild of a database index. > > > > > > Is there a better way I could be doing this? > > > > > > Curtis > > > > > > On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > I think you are trying to use the aliases wrongly. When you reindex and > > > > want to do "hot" replace of indices, you use a single alias pointing to a > > > > single index as the one the "client" uses. For example, have alias1 point > > > > to index1. Then, you reindex the data into index2, and once its done, you > > > > switch (in the same command) alias1 to point to index2 from index1. > > > > > > > > In the above usecase, there is no point where an alias is pointing to > > > > more than one index. Of course, you can have an alias point to more than > > > > one index, but it only really make sense when searching, and in this case, > > > > its the same as searching across several indices, where the index is part > > > > of the "uniqueness" of the document. > > > > > > > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > Ok, I'm going with the strategy of creating a new index, but I want to > > > > > do the reindexing all online using aliases. > > > > > > > > > > That leads to a couple of alias questions: > > > > > > > > > > 1. How are doc ids treated when you do a get (or search) operation on > > > > > an alias with multiple underlying indices? > > > > > 2. What happens if two docs with the same id exist in two of the > > > > > underlying indices? Is there some precedence or order to which doc is > > > > > returned? > > > > > 3. Does the index status API work with aliases? For example, can I > > > > > wait for yellow status on an alias rather than listing all the underlying > > > > > indices? > > > > > > > > > > Thanks again, > > > > > > > > > > Curtis > > > > > > > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > > > Ok, thanks. In that case I will go the route of indexing into a new > > > > > > index. > > > > > > > > > > > > Curtis > > > > > > > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > > > > > You can't change a field from not being indexed to being indexed. > > > > > > > You can change the default analyzer, but this will only affect future > > > > > > > documents indexed (by closing the index, updating the index settings, and > > > > > > > then opening it). > > > > > > > > > > > > > > If you end up reindexing the data, why not just index it into a new > > > > > > > index with the new mappings? > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone \<[caravone@gmail.com](mailto:caravone@gmail.com) > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > We are in a situation where we need to reindex a few hundred > > > > > > > > million docs (add some indexed fields, add some new fields). We hope to do > > > > > > > > this online by changing the mapping then performing updates on all the old > > > > > > > > docs to reindex them with the new mapping. > > > > > > > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed > > > > > > > > using the put mapping API? > > > > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > > > Curtis ---
**Author:** ![benny\_sadeh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benny_sadeh/32/2797_2.png) [@benny\_sadeh](https://discuss.elastic.co/u/benny_sadeh)\ **Post date:** [July 26, 2012, 3:44am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/11 "2012-07-26T03:44:32Z")
I can see two problems with your scheme: 1. what about a delete of a doc before it was reindexed? 2. what about a read of an updated doc before the reindexing is done? On Monday, July 23, 2012 9:07:38 PM UTC-10, Stephen Beeson wrote: > Hi Guys, > I found this thread was a useful starting point to handle my own simliar > issues. I did come up with another solution but wanted to run it by you to > see if you can spot any issues with it. > > The basic concept is to perform all reads of an index with one alias and > all writes with another, eg. index\_read and index\_write > When a reindex was performed, another index would be created and added to > the index\_write alias > > ``` > index_read -> old_index > > index_write -> old_index > new_index > > ``` > > This way all updates come through on the new index while it is being > populated and the old index continues to return newly updated documents. > > Once the reindexing is complete, I'd remove the old index and add the new > to the read alias, and remove the old index from the write alias. > Then I could delete the old and it should all be quite seemless. > > I have my own python code to access elasticsearch so the code that > references the two aliases in all in the one place. > > Does this seems like a reasonable approach? > > I think this could be made even easier if there was a mechanism similar to > the "search\_routing" and "index\_routing" features. Perhaps the aliases > could have add\_search and add\_index actions. Kimchy, does this sound like a > sensible feature? > > Thanks > Stephen > > On Wednesday, September 14, 2011 7:23:04 AM UTC+10, Curtis Caravone wrote: > > > Good point, I'll have to think about this some more. > > > > Curtis > > > > On Tue, Sep 13, 2011 at 2:15 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > Taking data while having the old index around can be problematic if you > > > have updates / deletes. You might need to still apply the changes to the > > > old index, and buffer them while you reindex to apply them again or > > > something like that. > > > > > > On Wed, Sep 14, 2011 at 12:02 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > Ok, I think that answers my questions. > > > > > > > > The reason I proposed using multiple indices with aliases is that I > > > > want to be able to take in new data while the migration is taking place. > > > > For example: > > > > > > > > index\_old: index with old mapping and data > > > > index\_new: index with new mapping > > > > > > > > At some point, switch writes to index\_new, so it starts filling up with > > > > new data. > > > > Then, migrate data from old to new while the system is online and still > > > > taking new data. > > > > > > > > I want to be able to search across both old and new during this > > > > process, similar to an online rebuild of a database index. > > > > > > > > Is there a better way I could be doing this? > > > > > > > > Curtis > > > > > > > > On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > I think you are trying to use the aliases wrongly. When you reindex > > > > > and want to do "hot" replace of indices, you use a single alias pointing to > > > > > a single index as the one the "client" uses. For example, have alias1 point > > > > > to index1. Then, you reindex the data into index2, and once its done, you > > > > > switch (in the same command) alias1 to point to index2 from index1. > > > > > > > > > > In the above usecase, there is no point where an alias is pointing to > > > > > more than one index. Of course, you can have an alias point to more than > > > > > one index, but it only really make sense when searching, and in this case, > > > > > its the same as searching across several indices, where the index is part > > > > > of the "uniqueness" of the document. > > > > > > > > > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > > > Ok, I'm going with the strategy of creating a new index, but I want > > > > > > to do the reindexing all online using aliases. > > > > > > > > > > > > That leads to a couple of alias questions: > > > > > > > > > > > > 1. How are doc ids treated when you do a get (or search) operation > > > > > > on an alias with multiple underlying indices? > > > > > > 2. What happens if two docs with the same id exist in two of the > > > > > > underlying indices? Is there some precedence or order to which doc is > > > > > > returned? > > > > > > 3. Does the index status API work with aliases? For example, can I > > > > > > wait for yellow status on an alias rather than listing all the underlying > > > > > > indices? > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > Curtis > > > > > > > > > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone \<[caravone@gmail.com](mailto:caravone@gmail.com) > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Ok, thanks. In that case I will go the route of indexing into a new > > > > > > > index. > > > > > > > > > > > > > > Curtis > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)wrote: > > > > > > > > > > > > > > > You can't change a field from not being indexed to being indexed. > > > > > > > > You can change the default analyzer, but this will only affect future > > > > > > > > documents indexed (by closing the index, updating the index settings, and > > > > > > > > then opening it). > > > > > > > > > > > > > > > > If you end up reindexing the data, why not just index it into a new > > > > > > > > index with the new mappings? > > > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone \< > > > > > > > > [caravone@gmail.com](mailto:caravone@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > We are in a situation where we need to reindex a few hundred > > > > > > > > > million docs (add some indexed fields, add some new fields). We hope to do > > > > > > > > > this online by changing the mapping then performing updates on all the old > > > > > > > > > docs to reindex them with the new mapping. > > > > > > > > > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed > > > > > > > > > using the put mapping API? > > > > > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > > > > > Curtis ---
**Author:** ![Chris\_Male](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_male/32/2607_2.png) [@Chris\_Male](https://discuss.elastic.co/u/Chris_Male)\ **Post date:** [July 26, 2012, 4:03am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/12 "2012-07-26T04:03:14Z")
Hi Stephen, I was wondering why have you chosen to separate reads and writes into separate indexes? Is this for performance? Were you seeing search performance degradation when indexing to the same index? On Tuesday, July 24, 2012 7:07:38 PM UTC+12, Stephen Beeson wrote: > Hi Guys, > I found this thread was a useful starting point to handle my own simliar > issues. I did come up with another solution but wanted to run it by you to > see if you can spot any issues with it. > > The basic concept is to perform all reads of an index with one alias and > all writes with another, eg. index\_read and index\_write > When a reindex was performed, another index would be created and added to > the index\_write alias > > ``` > index_read -> old_index > > index_write -> old_index > new_index > > ``` > > This way all updates come through on the new index while it is being > populated and the old index continues to return newly updated documents. > > Once the reindexing is complete, I'd remove the old index and add the new > to the read alias, and remove the old index from the write alias. > Then I could delete the old and it should all be quite seemless. > > I have my own python code to access elasticsearch so the code that > references the two aliases in all in the one place. > > Does this seems like a reasonable approach? > > I think this could be made even easier if there was a mechanism similar to > the "search\_routing" and "index\_routing" features. Perhaps the aliases > could have add\_search and add\_index actions. Kimchy, does this sound like a > sensible feature? > > Thanks > Stephen > > On Wednesday, September 14, 2011 7:23:04 AM UTC+10, Curtis Caravone wrote: > > > Good point, I'll have to think about this some more. > > > > Curtis > > > > On Tue, Sep 13, 2011 at 2:15 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > Taking data while having the old index around can be problematic if you > > > have updates / deletes. You might need to still apply the changes to the > > > old index, and buffer them while you reindex to apply them again or > > > something like that. > > > > > > On Wed, Sep 14, 2011 at 12:02 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > Ok, I think that answers my questions. > > > > > > > > The reason I proposed using multiple indices with aliases is that I > > > > want to be able to take in new data while the migration is taking place. > > > > For example: > > > > > > > > index\_old: index with old mapping and data > > > > index\_new: index with new mapping > > > > > > > > At some point, switch writes to index\_new, so it starts filling up with > > > > new data. > > > > Then, migrate data from old to new while the system is online and still > > > > taking new data. > > > > > > > > I want to be able to search across both old and new during this > > > > process, similar to an online rebuild of a database index. > > > > > > > > Is there a better way I could be doing this? > > > > > > > > Curtis > > > > > > > > On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > I think you are trying to use the aliases wrongly. When you reindex > > > > > and want to do "hot" replace of indices, you use a single alias pointing to > > > > > a single index as the one the "client" uses. For example, have alias1 point > > > > > to index1. Then, you reindex the data into index2, and once its done, you > > > > > switch (in the same command) alias1 to point to index2 from index1. > > > > > > > > > > In the above usecase, there is no point where an alias is pointing to > > > > > more than one index. Of course, you can have an alias point to more than > > > > > one index, but it only really make sense when searching, and in this case, > > > > > its the same as searching across several indices, where the index is part > > > > > of the "uniqueness" of the document. > > > > > > > > > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > > > Ok, I'm going with the strategy of creating a new index, but I want > > > > > > to do the reindexing all online using aliases. > > > > > > > > > > > > That leads to a couple of alias questions: > > > > > > > > > > > > 1. How are doc ids treated when you do a get (or search) operation > > > > > > on an alias with multiple underlying indices? > > > > > > 2. What happens if two docs with the same id exist in two of the > > > > > > underlying indices? Is there some precedence or order to which doc is > > > > > > returned? > > > > > > 3. Does the index status API work with aliases? For example, can I > > > > > > wait for yellow status on an alias rather than listing all the underlying > > > > > > indices? > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > Curtis > > > > > > > > > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone \<[caravone@gmail.com](mailto:caravone@gmail.com) > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Ok, thanks. In that case I will go the route of indexing into a new > > > > > > > index. > > > > > > > > > > > > > > Curtis > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)wrote: > > > > > > > > > > > > > > > You can't change a field from not being indexed to being indexed. > > > > > > > > You can change the default analyzer, but this will only affect future > > > > > > > > documents indexed (by closing the index, updating the index settings, and > > > > > > > > then opening it). > > > > > > > > > > > > > > > > If you end up reindexing the data, why not just index it into a new > > > > > > > > index with the new mappings? > > > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone \< > > > > > > > > [caravone@gmail.com](mailto:caravone@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > We are in a situation where we need to reindex a few hundred > > > > > > > > > million docs (add some indexed fields, add some new fields). We hope to do > > > > > > > > > this online by changing the mapping then performing updates on all the old > > > > > > > > > docs to reindex them with the new mapping. > > > > > > > > > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to indexed > > > > > > > > > using the put mapping API? > > > > > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > > > > > Curtis ---
**Author:** ![Stephen\_Beeson](https://avatars.discourse-cdn.com/v4/letter/s/58f4c7/32.png) [@Stephen\_Beeson](https://discuss.elastic.co/u/Stephen_Beeson)\ **Post date:** [July 26, 2012, 4:09am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/13 "2012-07-26T04:09:24Z")
No, I would only be using separate indexes while we were reindexing. This setup is to allow us to reindex safely in a live system, similar to Kimchys post: "When you reindex and want to do "hot" replace of indices, you use a single alias pointing to a single index as the one the "client" uses. For example, have alias1 point to index1. Then, you reindex the data into index2, and once its done, you switch (in the same command) alias1 to point to index2 from index1. " On Thursday, July 26, 2012 2:03:14 PM UTC+10, Chris Male wrote: > Hi Stephen, > > I was wondering why have you chosen to separate reads and writes into > separate indexes? Is this for performance? Were you seeing search > performance degradation when indexing to the same index? > > On Tuesday, July 24, 2012 7:07:38 PM UTC+12, Stephen Beeson wrote: > > > Hi Guys, > > I found this thread was a useful starting point to handle my own simliar > > issues. I did come up with another solution but wanted to run it by you to > > see if you can spot any issues with it. > > > > The basic concept is to perform all reads of an index with one alias and > > all writes with another, eg. index\_read and index\_write > > When a reindex was performed, another index would be created and added to > > the index\_write alias > > > > ``` > > index_read -> old_index > > > > index_write -> old_index > > new_index > > > > ``` > > > > This way all updates come through on the new index while it is being > > populated and the old index continues to return newly updated documents. > > > > Once the reindexing is complete, I'd remove the old index and add the new > > to the read alias, and remove the old index from the write alias. > > Then I could delete the old and it should all be quite seemless. > > > > I have my own python code to access elasticsearch so the code that > > references the two aliases in all in the one place. > > > > Does this seems like a reasonable approach? > > > > I think this could be made even easier if there was a mechanism similar > > to the "search\_routing" and "index\_routing" features. Perhaps the > > aliases could have add\_search and add\_index actions. Kimchy, does this > > sound like a sensible feature? > > > > Thanks > > Stephen > > > > On Wednesday, September 14, 2011 7:23:04 AM UTC+10, Curtis Caravone wrote: > > > > > Good point, I'll have to think about this some more. > > > > > > Curtis > > > > > > On Tue, Sep 13, 2011 at 2:15 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > Taking data while having the old index around can be problematic if you > > > > have updates / deletes. You might need to still apply the changes to the > > > > old index, and buffer them while you reindex to apply them again or > > > > something like that. > > > > > > > > On Wed, Sep 14, 2011 at 12:02 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > Ok, I think that answers my questions. > > > > > > > > > > The reason I proposed using multiple indices with aliases is that I > > > > > want to be able to take in new data while the migration is taking place. > > > > > For example: > > > > > > > > > > index\_old: index with old mapping and data > > > > > index\_new: index with new mapping > > > > > > > > > > At some point, switch writes to index\_new, so it starts filling up > > > > > with new data. > > > > > Then, migrate data from old to new while the system is online and > > > > > still taking new data. > > > > > > > > > > I want to be able to search across both old and new during this > > > > > process, similar to an online rebuild of a database index. > > > > > > > > > > Is there a better way I could be doing this? > > > > > > > > > > Curtis > > > > > > > > > > On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > > > I think you are trying to use the aliases wrongly. When you reindex > > > > > > and want to do "hot" replace of indices, you use a single alias pointing to > > > > > > a single index as the one the "client" uses. For example, have alias1 point > > > > > > to index1. Then, you reindex the data into index2, and once its done, you > > > > > > switch (in the same command) alias1 to point to index2 from index1. > > > > > > > > > > > > In the above usecase, there is no point where an alias is pointing to > > > > > > more than one index. Of course, you can have an alias point to more than > > > > > > one index, but it only really make sense when searching, and in this case, > > > > > > its the same as searching across several indices, where the index is part > > > > > > of the "uniqueness" of the document. > > > > > > > > > > > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > > > > > Ok, I'm going with the strategy of creating a new index, but I want > > > > > > > to do the reindexing all online using aliases. > > > > > > > > > > > > > > That leads to a couple of alias questions: > > > > > > > > > > > > > > 1. How are doc ids treated when you do a get (or search) operation > > > > > > > on an alias with multiple underlying indices? > > > > > > > 2. What happens if two docs with the same id exist in two of the > > > > > > > underlying indices? Is there some precedence or order to which doc is > > > > > > > returned? > > > > > > > 3. Does the index status API work with aliases? For example, can I > > > > > > > wait for yellow status on an alias rather than listing all the underlying > > > > > > > indices? > > > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > > > Curtis > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone \< > > > > > > > [caravone@gmail.com](mailto:caravone@gmail.com)\> wrote: > > > > > > > > > > > > > > > Ok, thanks. In that case I will go the route of indexing into a > > > > > > > > new index. > > > > > > > > > > > > > > > > Curtis > > > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)wrote: > > > > > > > > > > > > > > > > > You can't change a field from not being indexed to being indexed. > > > > > > > > > You can change the default analyzer, but this will only affect future > > > > > > > > > documents indexed (by closing the index, updating the index settings, and > > > > > > > > > then opening it). > > > > > > > > > > > > > > > > > > If you end up reindexing the data, why not just index it into a > > > > > > > > > new index with the new mappings? > > > > > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone \< > > > > > > > > > [caravone@gmail.com](mailto:caravone@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > We are in a situation where we need to reindex a few hundred > > > > > > > > > > million docs (add some indexed fields, add some new fields). We hope to do > > > > > > > > > > this online by changing the mapping then performing updates on all the old > > > > > > > > > > docs to reindex them with the new mapping. > > > > > > > > > > > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to > > > > > > > > > > indexed using the put mapping API? > > > > > > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > > > > > > > Curtis ---
**Author:** ![Stephen\_Beeson](https://avatars.discourse-cdn.com/v4/letter/s/58f4c7/32.png) [@Stephen\_Beeson](https://discuss.elastic.co/u/Stephen_Beeson)\ **Post date:** [July 26, 2012, 8:03am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/14 "2012-07-26T08:03:56Z")
Ok, so the detail that breaks all of this that I missed is: "It is an error to index to an alias which points to more than one index." Not sure why this is, but it ends this experiment. On Thursday, July 26, 2012 1:44:32 PM UTC+10, benny.sadeh wrote: > I can see two problems with your scheme: > > 1. what about a delete of a doc before it was reindexed? > 2. what about a read of an updated doc before the reindexing is done? > > On Monday, July 23, 2012 9:07:38 PM UTC-10, Stephen Beeson wrote: > > > Hi Guys, > > I found this thread was a useful starting point to handle my own simliar > > issues. I did come up with another solution but wanted to run it by you to > > see if you can spot any issues with it. > > > > The basic concept is to perform all reads of an index with one alias and > > all writes with another, eg. index\_read and index\_write > > When a reindex was performed, another index would be created and added to > > the index\_write alias > > > > ``` > > index_read -> old_index > > > > index_write -> old_index > > new_index > > > > ``` > > > > This way all updates come through on the new index while it is being > > populated and the old index continues to return newly updated documents. > > > > Once the reindexing is complete, I'd remove the old index and add the new > > to the read alias, and remove the old index from the write alias. > > Then I could delete the old and it should all be quite seemless. > > > > I have my own python code to access elasticsearch so the code that > > references the two aliases in all in the one place. > > > > Does this seems like a reasonable approach? > > > > I think this could be made even easier if there was a mechanism similar > > to the "search\_routing" and "index\_routing" features. Perhaps the > > aliases could have add\_search and add\_index actions. Kimchy, does this > > sound like a sensible feature? > > > > Thanks > > Stephen > > > > On Wednesday, September 14, 2011 7:23:04 AM UTC+10, Curtis Caravone wrote: > > > > > Good point, I'll have to think about this some more. > > > > > > Curtis > > > > > > On Tue, Sep 13, 2011 at 2:15 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > Taking data while having the old index around can be problematic if you > > > > have updates / deletes. You might need to still apply the changes to the > > > > old index, and buffer them while you reindex to apply them again or > > > > something like that. > > > > > > > > On Wed, Sep 14, 2011 at 12:02 AM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > Ok, I think that answers my questions. > > > > > > > > > > The reason I proposed using multiple indices with aliases is that I > > > > > want to be able to take in new data while the migration is taking place. > > > > > For example: > > > > > > > > > > index\_old: index with old mapping and data > > > > > index\_new: index with new mapping > > > > > > > > > > At some point, switch writes to index\_new, so it starts filling up > > > > > with new data. > > > > > Then, migrate data from old to new while the system is online and > > > > > still taking new data. > > > > > > > > > > I want to be able to search across both old and new during this > > > > > process, similar to an online rebuild of a database index. > > > > > > > > > > Is there a better way I could be doing this? > > > > > > > > > > Curtis > > > > > > > > > > On Tue, Sep 13, 2011 at 1:16 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > > > > > > > I think you are trying to use the aliases wrongly. When you reindex > > > > > > and want to do "hot" replace of indices, you use a single alias pointing to > > > > > > a single index as the one the "client" uses. For example, have alias1 point > > > > > > to index1. Then, you reindex the data into index2, and once its done, you > > > > > > switch (in the same command) alias1 to point to index2 from index1. > > > > > > > > > > > > In the above usecase, there is no point where an alias is pointing to > > > > > > more than one index. Of course, you can have an alias point to more than > > > > > > one index, but it only really make sense when searching, and in this case, > > > > > > its the same as searching across several indices, where the index is part > > > > > > of the "uniqueness" of the document. > > > > > > > > > > > > On Tue, Sep 13, 2011 at 9:05 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > > > > > > > > > Ok, I'm going with the strategy of creating a new index, but I want > > > > > > > to do the reindexing all online using aliases. > > > > > > > > > > > > > > That leads to a couple of alias questions: > > > > > > > > > > > > > > 1. How are doc ids treated when you do a get (or search) operation > > > > > > > on an alias with multiple underlying indices? > > > > > > > 2. What happens if two docs with the same id exist in two of the > > > > > > > underlying indices? Is there some precedence or order to which doc is > > > > > > > returned? > > > > > > > 3. Does the index status API work with aliases? For example, can I > > > > > > > wait for yellow status on an alias rather than listing all the underlying > > > > > > > indices? > > > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > > > Curtis > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 11:18 AM, Curtis Caravone \< > > > > > > > [caravone@gmail.com](mailto:caravone@gmail.com)\> wrote: > > > > > > > > > > > > > > > Ok, thanks. In that case I will go the route of indexing into a > > > > > > > > new index. > > > > > > > > > > > > > > > > Curtis > > > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 2:19 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)wrote: > > > > > > > > > > > > > > > > > You can't change a field from not being indexed to being indexed. > > > > > > > > > You can change the default analyzer, but this will only affect future > > > > > > > > > documents indexed (by closing the index, updating the index settings, and > > > > > > > > > then opening it). > > > > > > > > > > > > > > > > > > If you end up reindexing the data, why not just index it into a > > > > > > > > > new index with the new mappings? > > > > > > > > > > > > > > > > > > On Mon, Sep 12, 2011 at 3:00 AM, Curtis Caravone \< > > > > > > > > > [caravone@gmail.com](mailto:caravone@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > We are in a situation where we need to reindex a few hundred > > > > > > > > > > million docs (add some indexed fields, add some new fields). We hope to do > > > > > > > > > > this online by changing the mapping then performing updates on all the old > > > > > > > > > > docs to reindex them with the new mapping. > > > > > > > > > > > > > > > > > > > > Along these lines, I have a couple of questions: > > > > > > > > > > > > > > > > > > > > 1. Can a field mapping be changed from "index":"no" to to > > > > > > > > > > indexed using the put mapping API? > > > > > > > > > > 2. Can the default analyzer be changed with put mapping API? > > > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > > > > > > > Curtis ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:18am UTC](https://discuss.elastic.co/t/reindexing-with-new-mapping/5348/15 "2017-07-06T03:18:58Z")