Index corruption

Elastic Stack Elasticsearch
1

From time to time, my index gets corrupt and no longer returns the proper
facet information. I am running version 0.17.6 on Windows 7 64-bit. In order
to get my facet information to work the way I need it, I have created a
default-mapping.json file in the config folder with this in it:

{
"record": {
"dynamic_templates": [{
"index_nonanalyzed": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}]
}
}

I don't want any of my strings to be analyzed so they will be returned
"intact". For example, I want my facets to be like this "dirt and sand"
instead of it being broke up into three facets of "dirt", "and", and "sand".
I'm not sure if the dynamic templates approach is the best way to achieve
this but it seems to work.

However, when I stop ES and restart it by running elasticsearch.bat, I've
noticed that as long as I see the output say "re-syncing mappings with
cluster state for type [[record]]", the facets remain correct. If it doesn't
output that but only says "started", the facets return "exploded" into
separate pieces. Once this happens, it never goes back to having intact
facets; even if it says "re-syncing mappings...".

The only way to recover is to delete my data directory and reindex
everything.

Here are what the facets should look like:

https://lh5.googleusercontent.com/-oSIuDNVGgn8/ToYThMoWJdI/AAAAAAAAAY0/1n7MayzKC20/s400/facets_good.png

Here they are when they get messed up:

https://lh4.googleusercontent.com/-wwzbJrmyYGg/ToYToWrsaNI/AAAAAAAAAY4/WNbOzZrEqTQ/s400/facets_bad.png

Does anyone have any insight on what could be going on?

Thanks.

2

How many nodes do you have in the cluster? Also note that the
default-mapping config is wrong, the mapping type should not be "record", it
should be "default", but it might work for your case.

Its not really an index corruption, but it sounds like maybe something gets
wrong with the mappings, and replaying the transaction log on restart.

Can you create a simple recreation of this. Something like: Place this
mapping in default-mapping.json, start X nodes, run this curl requests,
shutdown the nodes, start them back up, and see the problem.

On Fri, Sep 30, 2011 at 10:08 PM, Keith Powell powellke@gmail.com wrote:

From time to time, my index gets corrupt and no longer returns the proper
facet information. I am running version 0.17.6 on Windows 7 64-bit. In order
to get my facet information to work the way I need it, I have created a
default-mapping.json file in the config folder with this in it:

{
"record": {
"dynamic_templates": [{
"index_nonanalyzed": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}]
}
}

I don't want any of my strings to be analyzed so they will be returned
"intact". For example, I want my facets to be like this "dirt and sand"
instead of it being broke up into three facets of "dirt", "and", and "sand".
I'm not sure if the dynamic templates approach is the best way to achieve
this but it seems to work.

However, when I stop ES and restart it by running elasticsearch.bat, I've
noticed that as long as I see the output say "re-syncing mappings with
cluster state for type [[record]]", the facets remain correct. If it doesn't
output that but only says "started", the facets return "exploded" into
separate pieces. Once this happens, it never goes back to having intact
facets; even if it says "re-syncing mappings...".

The only way to recover is to delete my data directory and reindex
everything.

Here are what the facets should look like:

https://lh5.googleusercontent.com/-oSIuDNVGgn8/ToYThMoWJdI/AAAAAAAAAY0/1n7MayzKC20/s400/facets_good.png

Here they are when they get messed up:

https://lh4.googleusercontent.com/-wwzbJrmyYGg/ToYToWrsaNI/AAAAAAAAAY4/WNbOzZrEqTQ/s400/facets_bad.png

Does anyone have any insight on what could be going on?

Thanks.

3

I have using only 1 node. I made the change to the default mapping config. I
thought I could specify my type in the default mapping since it's the only
one where I don't want strings analyzed.

I haven't been able to reproduce this issue today. It's possible it only
occurs after it runs for a longer period of time. I'll see what happens
tomorrow.

I just switched to version 0.17.7 and noticed that Ctrl+C to terminate the
batch file is working again. Could the problem be related to ES not shutting
down gracefully when I simply closed the command prompt window before?

4

I just switched to version 0.17.7 and noticed that Ctrl+C to terminate the
batch file is working again. Could the problem be related to ES not shutting
down gracefully when I simply closed the command prompt window before?

No, elasticsearch support a "kill -9" behavior and you don't have to shut it
down gracefully.

5

I haven't been able to reliably recreate this situation but it happened
again this morning. Last night, I put my laptop to sleep (with ES still
running). When I got to work, I docked my laptop, started it, and it locked
up. The lockup isn't ES related... it just locks up from time to time when I
put it in the docking station. Anyway, after I forced a restart of the
computer, I restarted ES and noticed that it was showing records that I had
deleted a few days ago and the facets were messed up as in the screenshots I
posted earlier.

I used ElasticSearch Head (http://mobz.github.com/elasticsearch-head/) to
look at it and it showed two nodes running but with different IP address; my
current IP address and my previous IP address. I typically only run one
node.

Does any of this make sense? Luckily, we haven't seen this issue on our
production server just my dev laptop.

Thanks.