Index corruption in ElasticSearch

Elastic Stack Elasticsearch
1

Hi

We reach a state where our ElasticSearch has some erratic behaviors. It
happened twice (I'm not sure the two were occurrences of the same problem
but the symptom were similar):

  • The first time it happens on my development machine with the very
    default configuration. At that time it was probably version 0.15. I was
    doing extensive schemas modifications, not deleting my index between the
    changes.
  • The second time is very recent. We have a cluster of two hosts and
    this is version 0.17.2. This time, the schemas did not change but we were
    tweaking the server settings a lot to have the cluster working.

We index some documents according to a schema with some fields like this
(sorry for the ruby syntax):

 "fullname"=>{"type"=>"string", "store" => true, "boost" => 1.2,

"include_in_all" => true},

and with those general settings:
"_source" => {"enabled" => false},
"_all" => {"enabled" => true},

When searching, we do queries like this

curl -XGET
'http://127.0.0.1:9200/picor/users/_search?q=suzan&fields=*&pretty=true'

We reached a state where Elasticsearch would not give complete answers. The
results are correct but the stored fields are not returned anymore.
Moreover, even after a deletion of the whole index and the recreation of the
index+schema, we were still in this state.
We saved the on disk state of elasticsearch and performed a full restart.
And after that, a deletion of the index would bring us back to a clean state
(Stored fields are return on search requests).

We kept a copy of the on disk files of elasticsearch at the time we were in
a bogus state. It might be of some interest for a post-mortem analysis. If
we relaunch ElasticSearch with those files and do queries about the current
schema, we get something like this:

curl -XGET 'http://127.0.0.1:9200/picor/users/_mapping?pretty=true'

{
"users" : {
"properties" : {
[...]
"fullname" : {
"include_in_all" : true,
"boost" : 1.2,
"type" : "string"
}
}
}
}

instead of this:

{
"users" : {
"_source" : {
"enabled" : false
},
"properties" : {
[...]
"fullname" : {
"include_in_all" : true,
"store" : "yes",
"boost" : 1.2,
"type" : "string"
}
}
}
}

Elasticsearch apparently lost that the '_source' should be deactivated and
that some fields must be stored. And this explain the search behavior.
Now, if we try to update the schema to the correct value, it does not change
anything. We must delete the whole index to return to a normal state. IIRC
I read somewhere that you can't update a schema and change a field from
'stored=false' to 'stored=true'. This is an annoying limitation but that
would explain that last observation.

Do you need additional informations for a bug report?
Thanks for your help

2

So the index does not get corrupted, but you end up with an index
configuration / mappings that you did not expect? The strange thing is that
its obviously is a custom configuration in elasticsearch (and not dynamic
mappings), so somehow it got there. Are you sure you correctly configure the
index created / mappings?

If you want, you can mail me a link to download the "corrupted" data, and I
can have a look.

On Fri, Aug 5, 2011 at 1:31 PM, freddj fastphilg@gmail.com wrote:

Hi

We reach a state where our Elasticsearch has some erratic behaviors. It
happened twice (I'm not sure the two were occurrences of the same problem
but the symptom were similar):

  • The first time it happens on my development machine with the very
    default configuration. At that time it was probably version 0.15. I was
    doing extensive schemas modifications, not deleting my index between the
    changes.
  • The second time is very recent. We have a cluster of two hosts and
    this is version 0.17.2. This time, the schemas did not change but we were
    tweaking the server settings a lot to have the cluster working.

We index some documents according to a schema with some fields like this
(sorry for the ruby syntax):

 "fullname"=>{"type"=>"string", "store" => true, "boost" => 1.2,

"include_in_all" => true},

and with those general settings:
"_source" => {"enabled" => false},
"_all" => {"enabled" => true},

When searching, we do queries like this

curl -XGET '
http://127.0.0.1:9200/picor/users/_search?q=suzan&fields=*&pretty=true'

We reached a state where Elasticsearch would not give complete answers.
The results are correct but the stored fields are not returned anymore.
Moreover, even after a deletion of the whole index and the recreation of the
index+schema, we were still in this state.
We saved the on disk state of elasticsearch and performed a full restart.
And after that, a deletion of the index would bring us back to a clean
state (Stored fields are return on search requests).

We kept a copy of the on disk files of elasticsearch at the time we were in
a bogus state. It might be of some interest for a post-mortem analysis. If
we relaunch Elasticsearch with those files and do queries about the current
schema, we get something like this:

curl -XGET 'http://127.0.0.1:9200/picor/users/_mapping?pretty=true'

{
"users" : {
"properties" : {
[...]
"fullname" : {
"include_in_all" : true,
"boost" : 1.2,
"type" : "string"
}
}
}
}

instead of this:

{
"users" : {
"_source" : {
"enabled" : false
},
"properties" : {
[...]
"fullname" : {
"include_in_all" : true,
"store" : "yes",
"boost" : 1.2,
"type" : "string"
}
}
}
}

Elasticsearch apparently lost that the '_source' should be deactivated and
that some fields must be stored. And this explain the search behavior.
Now, if we try to update the schema to the correct value, it does not
change anything. We must delete the whole index to return to a normal
state. IIRC I read somewhere that you can't update a schema and change a
field from 'stored=false' to 'stored=true'. This is an annoying limitation
but that would explain that last observation.

Do you need additional informations for a bug report?
Thanks for your help