- When i restarted a node yesterday, this happened (logs follow)-
- i lost 96 days worth of my logs. df before and after the restart
indicated that i had lost ~260G of indices post install. - After this, the node started syncing logs from the very beginning from
the rest of the cluster. - Specs- Elasticsearch 1.1.1 on Amazon aws. i use aws-cloud plugin for
node discovery.
What could have caused this?
[2014-05-29 13:35:37,824][INFO ][node ] [Pyro]
version[1.1.1], pid[4005], build[f1585f0/2014-04-16T14:27:12Z]
[2014-05-29 13:35:37,824][INFO ][node ] [Pyro]
initializing ...
[2014-05-29 13:35:37,851][INFO ][plugins ] [Pyro] loaded
[river-jdbc, cloud-aws], sites [whatson, head, bigdesk, paramedic]
[2014-05-29 13:35:41,166][INFO ][node ] [Pyro]
initialized
[2014-05-29 13:35:41,167][INFO ][node ] [Pyro] starting
...
[2014-05-29 13:35:41,306][INFO ][transport ] [Pyro]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.0.0.135:9300]}
[2014-05-29 13:35:45,802][INFO ][cluster.service ] [Pyro]
new_master
[Pyro][sWs8aTYHSx-j-ywO45vQ1Q][ip-10-0-0-135][inet[/10.0.0.135:9300]],
reason: zen-disco-join (elected_as_master)
[2014-05-29 13:35:45,826][INFO ][discovery ] [Pyro]
jarvis/sWs8aTYHSx-j-ywO45vQ1Q
[2014-05-29 13:35:46,019][INFO ][http ] [Pyro]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.0.0.135:9200]}
[2014-05-29 13:35:47,328][INFO ][gateway ] [Pyro] recovered
[96] indices into cluster_state
[2014-05-29 13:35:47,335][INFO ][node ] [Pyro] started
[2014-05-29 13:36:03,102][INFO ][node ] [Pyro] stopping
...
[2014-05-29 13:36:03,401][WARN ][cluster.service ] [Pyro] failed
to apply updated cluster state:
version [109], source [shard-started ([logstash-2014.02.27][4],
node[sWs8aTYHSx-j-ywO45vQ1Q], [P], s[INITIALIZING]), reason [after recovery
from gateway]]
nodes:
[Pyro][sWs8aTYHSx-j-ywO45vQ1Q][ip-10-0-0-135][inet[/10.0.0.135:9300]],
local, master
routing_table:
-- index [logstash-2014.03.22]
----shard_id [logstash-2014.03.22][4]
--------[logstash-2014.03.22][4], node[sWs8aTYHSx-j-ywO45vQ1Q], [P],
s[STARTED]
--------[logstash-2014.03.22][4], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.22][0]
--------[logstash-2014.03.22][0], node[null], [P], s[UNASSIGNED]
--------[logstash-2014.03.22][0], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.22][3]
--------[logstash-2014.03.22][3], node[null], [P], s[UNASSIGNED]
--------[logstash-2014.03.22][3], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.22][1]
--------[logstash-2014.03.22][1], node[sWs8aTYHSx-j-ywO45vQ1Q], [P],
s[STARTED]
--------[logstash-2014.03.22][1], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.22][2]
--------[logstash-2014.03.22][2], node[sWs8aTYHSx-j-ywO45vQ1Q], [P],
s[STARTED]
--------[logstash-2014.03.22][2], node[null], [R], s[UNASSIGNED]
-- index [logstash-2014.03.21]
----shard_id [logstash-2014.03.21][2]
--------[logstash-2014.03.21][2], node[sWs8aTYHSx-j-ywO45vQ1Q], [P],
s[STARTED]
--------[logstash-2014.03.21][2], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.21][0]
--------[logstash-2014.03.21][0], node[null], [P], s[UNASSIGNED]
--------[logstash-2014.03.21][0], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.21][3]
--------[logstash-2014.03.21][3], node[sWs8aTYHSx-j-ywO45vQ1Q], [P],
s[STARTED]
--------[logstash-2014.03.21][3], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.21][1]
--------[logstash-2014.03.21][1], node[null], [P], s[UNASSIGNED]
--------[logstash-2014.03.21][1], node[null], [R], s[UNASSIGNED]
----shard_id [logstash-2014.03.21][4]
--------[logstash-2014.03.21][4], node[sWs8aTYHSx-j-ywO45vQ1Q], [P],
s[STARTED]
--------[logstash-2014.03.21][4], node[null], [R], s[UNASSIGNED]
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/66b1c0f6-400c-4d57-82e9-c1704cc5ade4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.