Corrupt Cluster State


(Joshua Montgomery) #1

Is it possible to fix the state file if its been corrupted. Somehow a bad
mapping was added and it had an extra '}' throwing the entire state file
off. Deleting the bad mapping didn't fix the issue. Do I need to rebuild
the entire cluster or is there a quick fix? Should Elasticsearch even allow
bad mappings?

Also is it a good idea to run the cluster with this issue.

Thanks,
-Josh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6802bfc9-2b2e-4d44-9434-5c79e56462df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Boaz Leskes) #2

Hi Josh,

Creating a broken mapping shouldn't be possible and sounds like a serious
bug. Can you gist your broken mapping ? I would like to see if I can figure
out how it happened (if you know it will be great). Also - what version of
ES are you using?

I hope that seeing the mapping I can up with a work around. There are a
couple of other options but choosing one depends on the amount data you
have and wether you're OK with some down time.

Cheers,
Boaz

On Thursday, August 28, 2014 11:55:04 PM UTC+2, Joshua Montgomery wrote:

Is it possible to fix the state file if its been corrupted. Somehow a bad
mapping was added and it had an extra '}' throwing the entire state file
off. Deleting the bad mapping didn't fix the issue. Do I need to rebuild
the entire cluster or is there a quick fix? Should Elasticsearch even allow
bad mappings?

Also is it a good idea to run the cluster with this issue.

Thanks,
-Josh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/40f587b5-35c8-428e-a68c-bebacc843637%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Joshua Montgomery) #3

Thanks for the reply Boaz. Sorry for the late reply, was away for a while.
We run ES ver 1.2.2 and use the zulu openJDK from azul.

So I'm going to get you the mappings in another post, but I think the
cluster ended up becoming corrupt from an incident we saw about 1 1/2 weeks
ago. I started another discussion about it
here: https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/JwM0Gz9YO5M

I'm thinking the continuous updating of mappings (which caused the internal
DOS on the cluster) possibly had a sequence where it lead to a bad update
of the state file.

The state file for that incident was pretty big (70MB+), I'm not sure if
you want that. I can probably get a scoped down version of the mappings we
had after the incident.

Thanks,
-Josh

On Saturday, August 30, 2014 11:23:21 AM UTC-7, Boaz Leskes wrote:

Hi Josh,

Creating a broken mapping shouldn't be possible and sounds like a serious
bug. Can you gist your broken mapping ? I would like to see if I can figure
out how it happened (if you know it will be great). Also - what version of
ES are you using?

I hope that seeing the mapping I can up with a work around. There are a
couple of other options but choosing one depends on the amount data you
have and wether you're OK with some down time.

Cheers,
Boaz

On Thursday, August 28, 2014 11:55:04 PM UTC+2, Joshua Montgomery wrote:

Is it possible to fix the state file if its been corrupted. Somehow a bad
mapping was added and it had an extra '}' throwing the entire state file
off. Deleting the bad mapping didn't fix the issue. Do I need to rebuild
the entire cluster or is there a quick fix? Should Elasticsearch even allow
bad mappings?

Also is it a good idea to run the cluster with this issue.

Thanks,
-Josh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03ee610a-bc90-405d-8153-736e7e793f7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4