Corrupted ElasticSearch index?

Elastic Stack Elasticsearch
9

No, when things are running everything is ok, indexes break during
restart/powerdown
17-03-2014 13:11, "Clinton Gormley" clint@traveljury.com napisał(a):

Are you sure you didn't run out of disk space or file handles at some
stage, or have an OOM exception?

On 16 March 2014 16:37, bizzorama bizzorama@gmail.com wrote:

Hi,

it turned out that it was not a problem of ES version (we tested on both
0.90.10 and 0.90.9) but just a ES bug ...
after restarting pc or even just the service indices got broken ... we
found out that this was the case of missing mappings.
We observed that broken indices had their mappings corrupted (only some
default fields were observed).
You can check this by calling: http:\es_address:9200\indexName_mapping

Our mappings were dynamic (not set manually - just figured out by ES when
the records were incoming).

The solution was to add a static mapping file like the one described here:

Elasticsearch Platform — Find real-time answers at scale | Elastic(we added the default one).

I just copied mappings from a healty index, made some changes, turned it
to a mapping file and copied to the ES server.

Now everything works just fine.

Regards,
Karol

W dniu niedziela, 16 marca 2014 14:54:00 UTC+1 użytkownik Mac Jouz
napisał:

Hi Bizzorama,

I had a similar problem with the same configuration than you gave.
ES ran since the 11th of February and was fed every day at 6:00 AM by 2
LS.
Everything worked well (kibana reports were correct and no data loss)
until
I restarted yesterday ES :frowning:
Among 30 index (1 per day), 4 were unusable and data within kibana report
for the related period were unavailable (same org.elasticsearch.search.
facet.FacetPhaseExecutionException: Facet[0]: (key) field [@timestamp]
not found)

Do you confirm when you downgraded ES to 0.90.9 that you retrieved your
data
(i.e you was able to show your data in kibana reports) ?

I will try to downgrade ES version as you suggested and will let you know
more

Thanks for your answer

Sorry for the delay.

Looks like you were right, after downgrading ES to 0.90.9 i couldn't
reproduce the issue in such manner.

Unfortunately, I found some other problems, and one looks like a blocker
....

After whole ES cluster powerdown, ES just started replaying 'no mapping
for ... ' for each request.

W dniu czwartek, 20 lutego 2014 16:42:20 UTC+1 użytkownik Binh Ly
napisał:

Your error logs seem to indicate some kind of version mismatch. Is it
possible for you to test LS 1.3.2 against ES 0.90.9 and take a sample of
raw logs from those 3 days and test them through to see if those 3 days
work in Kibana? The reason I ask is because LS 1.3.2 (specifically the
elasticsearch output) was built using the binaries from ES 0.90.9.

Thanks.

Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit :

Hi,

I've noticed a very disturbing Elasticsearch behaviour ...
my environment is:

1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch
(0.90.10) + kibana

which process about 7 000 000 records per day,
everything worked fine on our test environment, untill we run some
tests for a longer period (about 15 days).

After that time, kibana was unable to show any data.
I did some investigation and it looks like some of the indexes (for 3
days to be exact) seem to be corrupted.
Now every query from kibana, using those corrupted indexes - failes.

Errors read from elasticsearch logs:

  • org.elasticsearch.search.facet.FacetPhaseExecutionException:
    Facet[terms]: failed to find mapping for Name* ... a couple of other
    columns*
  • org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet
    [0]: (key) field [@timestamp] not found

... generaly all queries end with those errors

When elasticsearch is started we find something like this:

[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name]
Message not fully read (request) for [243445] and action
[cluster/nodeIndexCreated], resetting
[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name]
Message not fully read (request) for [249943] and action
[cluster/nodeIndexCreated], resetting
[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name]
Message not fully read (request) for [246740] and action
[cluster/nodeIndexCreated], resetting

And a little observations:

  1. When using elasticsearch-head plugin, when querying records
    'manually', i can see only elasticsearch columns (_index, _type, _id,
    _score).
    But when I 'randomly' select columns and overview their raw json
    they look ok.

2, When I tried to process same data again - everything is ok

Is it possible that some corrupted data found its way to elasticsearch
and now whole index is broken ?
Can this be fixed ? reindexed or sth ?
This data is very importand and can't be lost ...

Best Regards,
Karol

Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit :

Hi,

I've noticed a very disturbing Elasticsearch behaviour ...
my environment is:

1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch
(0.90.10) + kibana

which process about 7 000 000 records per day,
everything worked fine on our test environment, untill we run some
tests for a longer period (about 15 days).

After that time, kibana was unable to show any data.
I did some investigation and it looks like some of the indexes (for 3
days to be exact) seem to be corrupted.
Now every query from kibana, using those corrupted indexes - failes.

Errors read from elasticsearch logs:

  • org.elasticsearch.search.facet.FacetPhaseExecutionException:
    Facet[terms]: failed to find mapping for Name* ... a couple of other
    columns*
  • org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet
    [0]: (key) field [@timestamp] not found

... generaly all queries end with those errors

When elasticsearch is started we find something like this:

[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name]
Message not fully read (request) for [243445] and action
[cluster/nodeIndexCreated], resetting
[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name]
Message not fully read (request) for [249943] and action
[cluster/nodeIndexCreated], resetting
[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name]
Message not fully read (request) for [246740] and action
[cluster/nodeIndexCreated], resetting

And a little observations:

  1. When using elasticsearch-head plugin, when querying records
    'manually', i can see only elasticsearch columns (_index, _type, _id,
    _score).
    But when I 'randomly' select columns and overview their raw json
    they look ok.

2, When I tried to process same data again - everything is ok

Is it possible that some corrupted data found its way to elasticsearch
and now whole index is broken ?
Can this be fixed ? reindexed or sth ?
This data is very importand and can't be lost ...

Best Regards,
Karol

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/7ZwB6SNFkDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPt3XKSE6%3DX-QjwJYzs0Aq7FXOHbE6Vu8Am-5BGvznmAsbFikw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPt3XKSE6%3DX-QjwJYzs0Aq7FXOHbE6Vu8Am-5BGvznmAsbFikw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMSRqaaudS6_xZGiJojLe5Ccr7BpEcJOzuY_39ffZ3m9ry_htQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.