I've been trying to trouble shoot an issue with my single ES node.
When I went to go look at it, it was at 100% diskspace usage. A lot of this
issue was due to ES logs taking up space on the volume.
When I cleared out the logs and recovered a lot of the space and tried to
restart then I saw a failed shard issue:
[INFO ][node ] [Bloodshed] stopping ...
[2014-03-20 19:39:26,957][WARN ][cluster.action.shard ] [Bloodshed]
sending failed shard for [logstash-2014.02.06][4],
node[ap2Lm43gRIyoN0FDh-ZO_A], [R], s[INITIALIZING], reason [Failed to start
shard, message [RecoveryFailedException[[logstash-2014.02.06][4]: Recovery
failed from [Forbush Man][x5RvlhLLRPOtz_hcgW-JgA][inet[/10.0.30.55:9300]]
into [Bloodshed][ap2Lm43gRIyoN0FDh-ZO_A][inet[/10.0.30.50:9300]]]; nested:
RemoteTransportException[[Forbush
Man][inet[/10.0.30.55:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[logstash-2014.02.06][4] Phase[2] Execution
failed]; nested:
RemoteTransportException[[Bloodshed][inet[/10.0.30.50:9300]][index/shard/recovery/prepareTranslog]];
nested: EngineCreationFailureException[[logstash-2014.02.06][4] failed to
create engine]; nested: IndexFormatTooNewException[Format version is not
supported (resource:
ChecksumIndexInput(NIOFSIndexInput(path="/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2014.02.06/4/index/segments_3"))):
1 (needs to be between 0 and 0)]; ]]
[2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed]
stopped
[2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed]
closing ...
[2014-03-20 19:39:28,119][INFO ][node ] [Bloodshed]
closed
I had changed the config file to
discovery.zen.ping.multicast.enabled: false
and tried restarting the server - no more entries in the log
I also tried moving some of the data (especially that corrupted entry) from
the data directory and restarting. The system message from init.d would say
that ES had started, but then checking the status again it would stay that
it was stopped.
Again no more log entries.
I tried renaming the log file and then touching a file called
elasticsearch.log and chowing it to elasticsearch:elasticsearch | a restart
of the service, it still wasn't running and no log entries
At this point I decided to upgrade my 0.90 ES server to the new 1.0 ES with
a .deb package
I installed, rebooted the machine and after restarting the service - no
entries
My last try was to make a new data directory, a clean data directory, and
then point to that in the config file and restart the ES server.
It still would not start and nothing in the log.
At this point I'm not sure what else to troubleshoot on how to get it back
working again. The data I'm not as worried about, but I need the Elastic
Search server up and running again.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3e438abe-6fed-4333-9155-f6c8e7e2c040%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.