# Index corruption in ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [August 5, 2011, 10:31am UTC](https://discuss.elastic.co/t/index-corruption-in-elasticsearch/5063 "2011-08-05T10:31:59Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![freddj](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@freddj](https://discuss.elastic.co/u/freddj)\ **Post date:** [August 5, 2011, 10:31am UTC](https://discuss.elastic.co/t/index-corruption-in-elasticsearch/5063/1 "2011-08-05T10:31:59Z")
Hi We reach a state where our ElasticSearch has some erratic behaviors. It happened twice (I'm not sure the two were occurrences of the same problem but the symptom were similar): - The first time it happens on my development machine with the very default configuration. At that time it was probably version 0.15. I was doing extensive schemas modifications, not deleting my index between the changes. - The second time is very recent. We have a cluster of two hosts and this is version 0.17.2. This time, the schemas did not change but we were tweaking the server settings a lot to have the cluster working. We index some documents according to a schema with some fields like this (sorry for the ruby syntax): ``` "fullname"=>{"type"=>"string", "store" => true, "boost" => 1.2, ``` "include\_in\_all" =\> true}, and with those general settings: "\_source" =\> {"enabled" =\> false}, "\_all" =\> {"enabled" =\> true}, When searching, we do queries like this curl -XGET '[http://127.0.0.1:9200/picor/users/\_search?q=suzan&fields=\*&pretty=true](http://127.0.0.1:9200/picor/users/_search?q=suzan&fields=*&pretty=true)' We reached a state where Elasticsearch would not give complete answers. The results are correct but the stored fields are not returned anymore. Moreover, even after a deletion of the whole index and the recreation of the index+schema, we were still in this state. We saved the on disk state of elasticsearch and performed a full restart. And after that, a deletion of the index would bring us back to a clean state (Stored fields are return on search requests). We kept a copy of the on disk files of elasticsearch at the time we were in a bogus state. It might be of some interest for a post-mortem analysis. If we relaunch ElasticSearch with those files and do queries about the current schema, we get something like this: curl -XGET '[http://127.0.0.1:9200/picor/users/\_mapping?pretty=true](http://127.0.0.1:9200/picor/users/_mapping?pretty=true)' { "users" : { "properties" : { [...] "fullname" : { "include\_in\_all" : true, "boost" : 1.2, "type" : "string" } } } } instead of this: { "users" : { "\_source" : { "enabled" : false }, "properties" : { [...] "fullname" : { "include\_in\_all" : true, "store" : "yes", "boost" : 1.2, "type" : "string" } } } } Elasticsearch apparently lost that the '\_source' should be deactivated and that some fields must be stored. And this explain the search behavior. Now, if we try to update the schema to the correct value, it does not change anything. We must delete the whole index to return to a normal state. IIRC I read somewhere that you can't update a schema and change a field from 'stored=false' to 'stored=true'. This is an annoying limitation but that would explain that last observation. Do you need additional informations for a bug report? Thanks for your help ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 6, 2011, 5:28pm UTC](https://discuss.elastic.co/t/index-corruption-in-elasticsearch/5063/2 "2011-08-06T17:28:28Z")
So the index does not get corrupted, but you end up with an index configuration / mappings that you did not expect? The strange thing is that its obviously _is_ a custom configuration in elasticsearch (and not dynamic mappings), so somehow it got there. Are you sure you correctly configure the index created / mappings? If you want, you can mail me a link to download the "corrupted" data, and I can have a look. On Fri, Aug 5, 2011 at 1:31 PM, freddj [fastphilg@gmail.com](mailto:fastphilg@gmail.com) wrote: > Hi > > We reach a state where our Elasticsearch has some erratic behaviors. It > happened twice (I'm not sure the two were occurrences of the same problem > but the symptom were similar): > > - The first time it happens on my development machine with the very > default configuration. At that time it was probably version 0.15. I was > doing extensive schemas modifications, not deleting my index between the > changes. > - The second time is very recent. We have a cluster of two hosts and > this is version 0.17.2. This time, the schemas did not change but we were > tweaking the server settings a lot to have the cluster working. > > We index some documents according to a schema with some fields like this > (sorry for the ruby syntax): > > ``` > "fullname"=>{"type"=>"string", "store" => true, "boost" => 1.2, > > ``` > > "include\_in\_all" =\> true}, > > and with those general settings: > "\_source" =\> {"enabled" =\> false}, > "\_all" =\> {"enabled" =\> true}, > > When searching, we do queries like this > > curl -XGET ' > [http://127.0.0.1:9200/picor/users/\_search?q=suzan&fields=\*&pretty=true](http://127.0.0.1:9200/picor/users/_search?q=suzan&fields=*&pretty=true)' > > We reached a state where Elasticsearch would not give complete answers. > The results are correct but the stored fields are not returned anymore. > Moreover, even after a deletion of the whole index and the recreation of the > index+schema, we were still in this state. > We saved the on disk state of elasticsearch and performed a full restart. > And after that, a deletion of the index would bring us back to a clean > state (Stored fields are return on search requests). > > We kept a copy of the on disk files of elasticsearch at the time we were in > a bogus state. It might be of some interest for a post-mortem analysis. If > we relaunch Elasticsearch with those files and do queries about the current > schema, we get something like this: > > curl -XGET '[http://127.0.0.1:9200/picor/users/\_mapping?pretty=true](http://127.0.0.1:9200/picor/users/_mapping?pretty=true)' > > { > "users" : { > "properties" : { > [...] > "fullname" : { > "include\_in\_all" : true, > "boost" : 1.2, > "type" : "string" > } > } > } > } > > instead of this: > > { > "users" : { > "\_source" : { > "enabled" : false > }, > "properties" : { > [...] > "fullname" : { > "include\_in\_all" : true, > "store" : "yes", > "boost" : 1.2, > "type" : "string" > } > } > } > } > > Elasticsearch apparently lost that the '\_source' should be deactivated and > that some fields must be stored. And this explain the search behavior. > Now, if we try to update the schema to the correct value, it does not > change anything. We must delete the whole index to return to a normal > state. IIRC I read somewhere that you can't update a schema and change a > field from 'stored=false' to 'stored=true'. This is an annoying limitation > but that would explain that last observation. > > Do you need additional informations for a bug report? > Thanks for your help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:58am UTC](https://discuss.elastic.co/t/index-corruption-in-elasticsearch/5063/3 "2017-07-06T03:58:03Z")