# Mapping changes and their affect on indexed data **URL:** \ **Category:** Elasticsearch\ **Created:** [May 7, 2012, 5:46pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583 "2012-05-07T17:46:10Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [May 7, 2012, 5:46pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/1 "2012-05-07T17:46:10Z")
I'm using the S3 Gateway, but I think my use case is the same regardless of my gateway type. Suppose I shut down my running cluster, and make some rather complicated updates to my mapping files. When I start up my cluster again, what happens to my indicies? ---
**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\ **Post date:** [May 7, 2012, 5:53pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/2 "2012-05-07T17:53:07Z")
Hello! Your index structure won't be changed if that's what you ask about. New indices created will have new mappings. You can always try to update your existing mappings with [http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html,](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html) some changes (like adding new fields) can be done. _--_ Regards, Rafał Kuć Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch - ElasticSearch_ I'm using the S3 Gateway, but I think my use case is the same regardless of my gateway type. Suppose I shut down my running cluster, and make some rather complicated updates to my mapping files. When I start up my cluster again, what happens to my indicies? ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [May 7, 2012, 5:59pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/3 "2012-05-07T17:59:56Z")
Thanks. So what I am wondering is whether a feature might be added to recover by: 1. Access the \_source property from documents already indexed and in the gateway. 2. Reindex those documents using the new mappings A lot of us are using ES more as a database, and I'm having to write my '\_source' out to another form of permanent storage and then recovering in this manner when mapping files change or when a recovery fails for some reason. On Monday, May 7, 2012 1:53:07 PM UTC-4, Rafał Kuć wrote: > Hello! > > Your index structure won't be changed if that's what you ask about. New > indices created will have new mappings. You can always try to update your > existing mappings with > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html,)[http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html) some > changes (like adding new fields) can be done. > > \*-- > Regards, > Rafał Kuć > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - > Elasticsearch > > - > > I'm using the S3 Gateway, but I think my use case is the same regardless > of my gateway type. Suppose I shut down my running cluster, and make some > rather complicated updates to my mapping files. > > When I start up my cluster again, what happens to my indicies? ---
**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\ **Post date:** [May 7, 2012, 6:16pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/4 "2012-05-07T18:16:12Z")
Hello! In theory that would work, but think about the following - you change the type of the field you facet on from string to integer. Some of the documents in the index have integer field, while some have string. This may cause problems for your queries. Maybe something like this would be an opion - create an alias which you use for querying on the original index. Then when you need to change your mappings totally, you create a new index with those mappings, index the data once again and then just change the alias to point to the new index instead of the old one. Of course use of such scenario depends on the volume of your data and such. _--_ Regards, Rafał Kuć Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch - ElasticSearch_ Thanks. So what I am wondering is whether a feature might be added to recover by: 1. Access the \_source property from documents already indexed and in the gateway. 2. Reindex those documents using the new mappings A lot of us are using ES more as a database, and I'm having to write my '\_source' out to another form of permanent storage and then recovering in this manner when mapping files change or when a recovery fails for some reason. On Monday, May 7, 2012 1:53:07 PM UTC-4, Rafał Kuć wrote: Hello! Your index structure won't be changed if that's what you ask about. New indices created will have new mappings. You can always try to update your existing mappings with [http://www.elasticsearch.](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html)[org/guide/reference/api/admin-](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html)[indices-put-mapping.html,](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html) some changes (like adding new fields) can be done. _--_ Regards, Rafał Kuć Sematext :: [http://sematext.com/](http://sematext.com/) _:: Solr - Lucene - Nutch - ElasticSearch_ I'm using the S3 Gateway, but I think my use case is the same regardless of my gateway type. Suppose I shut down my running cluster, and make some rather complicated updates to my mapping files. When I start up my cluster again, what happens to my indicies? ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [May 7, 2012, 7:00pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/5 "2012-05-07T19:00:00Z")
On May 7, 10:59 am, James Cook [jc...@pykl.com](mailto:jc...@pykl.com) wrote: > Thanks. So what I am wondering is whether a feature might be added to > recover by: > > 1. Access the \_source property from documents already indexed and in the > gateway. > 2. Reindex those documents using the new mappings > > A lot of us are using ES more as a database, and I'm having to write my > '\_source' out to another form of permanent storage and then recovering in > this manner when mapping files change or when a recovery fails for some > reason. Support for schema migration would be great. I'm curious, how often have people here experienced recovery failures and have had to rebuild the data from an external source? ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [May 7, 2012, 7:51pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/6 "2012-05-07T19:51:35Z")
In that workflow I was suggesting, no documents would have a field with both string and integer types. The "old" index would have it as a String, but the "new" index would have it as an integer. It wouldn't cause a problem with querying data. That isn't to say that a migration of previously indexed documents wouldn't involve transformation. Any robust migration capability does. -- jim On Monday, May 7, 2012 2:16:12 PM UTC-4, Rafał Kuć wrote: > Hello! > > In theory that would work, but think about the following - you change the > type of the field you facet on from string to integer. Some of the > documents in the index have integer field, while some have string. This may > cause problems for your queries. Maybe something like this would be an > opion - create an alias which you use for querying on the original index. > Then when you need to change your mappings totally, you create a new index > with those mappings, index the data once again and then just change the > alias to point to the new index instead of the old one. Of course use of > such scenario depends on the volume of your data and such. > > \*-- > Regards, > Rafał Kuć > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - > Elasticsearch > > - > > Thanks. So what I am wondering is whether a feature might be added to > recover by: > > 1. Access the \_source property from documents already indexed and in > the gateway. > 2. Reindex those documents using the new mappings > > A lot of us are using ES more as a database, and I'm having to write my > '\_source' out to another form of permanent storage and then recovering in > this manner when mapping files change or when a recovery fails for some > reason. > > On Monday, May 7, 2012 1:53:07 PM UTC-4, Rafał Kuć wrote: > Hello! > > Your index structure won't be changed if that's what you ask about. New > indices created will have new mappings. You can always try to update your > existing mappings with [http://www.elasticsearch](http://www.elasticsearch).[http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html) > org/guide/reference/api/admin-[http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html) > indices-put-mapping.html,[http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html) some > changes (like adding new fields) can be done. > > \*-- > Regards, > Rafał Kuć > Sematext :: _[http://sematext.com/](http://sematext.com/)_ :: Solr - Lucene - Nutch - > Elasticsearch > > - > > I'm using the S3 Gateway, but I think my use case is the same regardless > of my gateway type. Suppose I shut down my running cluster, and make some > rather complicated updates to my mapping files. > > When I start up my cluster again, what happens to my indicies? ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [May 7, 2012, 7:56pm UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/7 "2012-05-07T19:56:18Z")
This isn't the most accurate list of people with these types of failures, but it has happened before. Shay has plugged many of the issues you might find on the old mailing list over time. [http://goo.gl/Gwo05](http://goo.gl/Gwo05) On Monday, May 7, 2012 3:00:00 PM UTC-4, Eric Jain wrote: > On May 7, 10:59 am, James Cook [jc...@pykl.com](mailto:jc...@pykl.com) wrote: > > > Thanks. So what I am wondering is whether a feature might be added to > > recover by: > > > > 1. Access the \_source property from documents already indexed and in > > the > > gateway. > > 2. Reindex those documents using the new mappings > > > > A lot of us are using ES more as a database, and I'm having to write my > > '\_source' out to another form of permanent storage and then recovering > > in > > this manner when mapping files change or when a recovery fails for some > > reason. > > Support for schema migration would be great. > > I'm curious, how often have people here experienced recovery failures > and have had to rebuild the data from an external source? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 9, 2012, 9:18am UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/8 "2012-05-09T09:18:15Z")
The gateway (s3 and local) actually store Lucene indices... . Note, there is a big difference between the shared gateway (s3) and local gateway, where the shared gateway always have a "single" copy of the data (on s3) and treats it as the master copy, and the local gateway uses each node storage to recovery (so replicas you have, the better, since it can recover from them as well). In order to have a better story around proper backups, which you can do with local gateway and backing up the data directory, I discussed previously a backup/restore API option, so you can backup a point in time state of an index (or the cluster), and then recover from it into a new index, or existing index. This while using the local gateway, and for example, doing point in time backups to s3. On AWS, I recommend using the local gateway and not the s3 gateway, because it has less resource overload on the system, and moves from the "single" copy of master data (on s3). On Mon, May 7, 2012 at 10:56 PM, James Cook [jcook@pykl.com](mailto:jcook@pykl.com) wrote: > This isn't the most accurate list of people with these types of failures, > but it has happened before. Shay has plugged many of the issues you might > find on the old mailing list over time. > > [Redirecting to Google Groups](http://goo.gl/Gwo05) > > On Monday, May 7, 2012 3:00:00 PM UTC-4, Eric Jain wrote: > > > On May 7, 10:59 am, James Cook [jc...@pykl.com](mailto:jc...@pykl.com) wrote: > > > > > Thanks. So what I am wondering is whether a feature might be added to > > > recover by: > > > > > > 1. Access the \_source property from documents already indexed and in > > > the > > > gateway. > > > 2. Reindex those documents using the new mappings > > > > > > A lot of us are using ES more as a database, and I'm having to write my > > > '\_source' out to another form of permanent storage and then recovering > > > in > > > this manner when mapping files change or when a recovery fails for some > > > reason. > > > > Support for schema migration would be great. > > > > I'm curious, how often have people here experienced recovery failures > > and have had to rebuild the data from an external source? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:29am UTC](https://discuss.elastic.co/t/mapping-changes-and-their-affect-on-indexed-data/7583/9 "2017-07-06T03:29:42Z")