I have an index that has 9,481,043 docs. It is a real pain in my backside
for the last few days.
Firstly this index has an alias. I restarted the servers last week and
after a few days noticed that after the restart there was a new index with
the same name as the alias and my data was going into this new index and
not the correct one.
That was odd enough but now I had to restart the cluster again today due to
problems with the same index and now it has deleted, from disk, 2 of the 4
shards. from both of my instances in the cluster.
It also deleted the index that it created with the alias name so I have now
lost some data completely.
Unfortunately this data only exists in ES so I am unable to re index the
data.
I had errors in the log file that look like this
[INFO ][gateway.local.state.meta ] [HOST_NAME] [INDEX_NAME] dangling index, exists on local file system, but not in cluster metadata, scheduling to delete in [2h], auto import to cluster state [YES]
These, below, happen on restarting an instance.
[2014-07-21 12:34:19,507][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:19,562][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:19,738][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:20,827][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:20,860][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:20,973][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:21,391][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:21,424][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:21,544][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:26,901][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:26,950][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:27,060][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:28,084][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:28,118][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:28,244][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:29,179][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:29,349][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:29,555][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:30,113][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:30,157][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:30,312][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:30,876][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:30,922][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:31,055][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:32,770][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:32,807][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [api], shards [4]/[1], mappings [] [2014-07-21 12:34:32,894][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic) [2014-07-21 12:34:33,388][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] deleting index [2014-07-21 12:34:33,420][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [] [2014-07-21 12:34:33,554][INFO ][cluster.metadata ] [i-3e83217e] [index_name_alias] update_mapping [search] (dynamic)
This all started because I was getting these errors, so I restarted the
cluster and it appeared to fix it but has caused no end of issues but only
with one index. I have about 30 index on this cluster
[2014-07-16 00:18:38,786][ERROR][indices.fielddata.breaker] [i-3e83217e] New used memory 2809492636 [2.6gb] from field [date_from] would be larger than configured breaker: 2809371033 [2.6gb], breaking </pre -- Follow us Like us Read our blog ·Get our newsletter ·Download our free iPhone app The information in this email and any files transmitted with it is confidential and may be legally privileged. This email is intended solely for the addressee. If you are not the intended recipient, please notify the sender by reply, confirming that you have deleted the email. Any disclosure, copying, or distribution of the email, or any action taken or failed to be taken in reliance on it by an unauthorised recipient is prohibited and may be unlawful. Recipients should kindly notify the sender if they no longer wish to communicate by email. Where the email is sent in a non-business capacity, then the email is not in any way associated with Smith Global, Mr & Mrs Smith (Asia Pacific) Pty Ltd or Mr & Mrs Smith (Americas) Inc nor sent on their behalf. Smith Global, Mr & Mrs Smith (Asia Pacific) Pty Ltd and Mr & Mrs Smith (Americas) Inc may monitor outgoing and incoming emails and other email attachments. Smith Global Ltd is registered in the UK, company number: 443 8845. Mr & Mrs Smith (Asia Pacific) Pty Ltd is registered in Australia, ABN 88 134 136 128. Mr & Mrs Smith (Americas) Inc is registered in the United States -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4f2f4fc0-26a6-41dc-bed4-d0aec8e6b421%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.