Elasticsearch not writing to logs on start


(Adam B) #1

I've been trying to trouble shoot an issue with my single ES node.

When I went to go look at it, it was at 100% diskspace usage. A lot of this
issue was due to ES logs taking up space on the volume.

When I cleared out the logs and recovered a lot of the space and tried to
restart then I saw a failed shard issue:

[INFO ][node ] [Bloodshed] stopping ...
[2014-03-20 19:39:26,957][WARN ][cluster.action.shard ] [Bloodshed]
sending failed shard for [logstash-2014.02.06][4],
node[ap2Lm43gRIyoN0FDh-ZO_A], [R], s[INITIALIZING], reason [Failed to start
shard, message [RecoveryFailedException[[logstash-2014.02.06][4]: Recovery
failed from [Forbush Man][x5RvlhLLRPOtz_hcgW-JgA][inet[/10.0.30.55:9300]]
into [Bloodshed][ap2Lm43gRIyoN0FDh-ZO_A][inet[/10.0.30.50:9300]]]; nested:
RemoteTransportException[[Forbush
Man][inet[/10.0.30.55:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[logstash-2014.02.06][4] Phase[2] Execution
failed]; nested:
RemoteTransportException[[Bloodshed][inet[/10.0.30.50:9300]][index/shard/recovery/prepareTranslog]];
nested: EngineCreationFailureException[[logstash-2014.02.06][4] failed to
create engine]; nested: IndexFormatTooNewException[Format version is not
supported (resource:
ChecksumIndexInput(NIOFSIndexInput(path="/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2014.02.06/4/index/segments_3"))):
1 (needs to be between 0 and 0)]; ]]
[2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed]
stopped
[2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed]
closing ...
[2014-03-20 19:39:28,119][INFO ][node ] [Bloodshed]
closed

I had changed the config file to

discovery.zen.ping.multicast.enabled: false

and tried restarting the server - no more entries in the log

I also tried moving some of the data (especially that corrupted entry) from
the data directory and restarting. The system message from init.d would say
that ES had started, but then checking the status again it would stay that
it was stopped.

Again no more log entries.

I tried renaming the log file and then touching a file called
elasticsearch.log and chowing it to elasticsearch:elasticsearch | a restart
of the service, it still wasn't running and no log entries

At this point I decided to upgrade my 0.90 ES server to the new 1.0 ES with
a .deb package
I installed, rebooted the machine and after restarting the service - no
entries

My last try was to make a new data directory, a clean data directory, and
then point to that in the config file and restart the ES server.

It still would not start and nothing in the log.

At this point I'm not sure what else to troubleshoot on how to get it back
working again. The data I'm not as worried about, but I need the Elastic
Search server up and running again.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3e438abe-6fed-4333-9155-f6c8e7e2c040%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Alexander Reelsen) #2

Hey,

can you check if the permissions are correctly set? Check for the
elasticsearch user in /etc/passwd and check if that user is allowed to
write into /var/log/elasticsearch and the data directories.

if this does not work or give any enlightenment, can you please open an
issue, so we can debug this further? Thanks!

--Alex

On Fri, Mar 21, 2014 at 3:15 AM, Adam B abuhmann.dba@gmail.com wrote:

I've been trying to trouble shoot an issue with my single ES node.

When I went to go look at it, it was at 100% diskspace usage. A lot of
this issue was due to ES logs taking up space on the volume.

When I cleared out the logs and recovered a lot of the space and tried to
restart then I saw a failed shard issue:

[INFO ][node ] [Bloodshed] stopping ...
[2014-03-20 19:39:26,957][WARN ][cluster.action.shard ] [Bloodshed]
sending failed shard for [logstash-2014.02.06][4],
node[ap2Lm43gRIyoN0FDh-ZO_A], [R], s[INITIALIZING], reason [Failed to start
shard, message [RecoveryFailedException[[logstash-2014.02.06][4]: Recovery
failed from [Forbush Man][x5RvlhLLRPOtz_hcgW-JgA][inet[/10.0.30.55:9300]]
into [Bloodshed][ap2Lm43gRIyoN0FDh-ZO_A][inet[/10.0.30.50:9300]]];
nested: RemoteTransportException[[Forbush Man][inet[/10.0.30.55:9300]][index/shard/recovery/startRecovery]];
nested: RecoveryEngineException[[logstash-2014.02.06][4] Phase[2] Execution
failed]; nested: RemoteTransportException[[Bloodshed][inet[/10.0.30.50:9300]][index/shard/recovery/prepareTranslog]];
nested: EngineCreationFailureException[[logstash-2014.02.06][4] failed to
create engine]; nested: IndexFormatTooNewException[Format version is not
supported (resource:
ChecksumIndexInput(NIOFSIndexInput(path="/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2014.02.06/4/index/segments_3"))):
1 (needs to be between 0 and 0)]; ]]
[2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed]
stopped
[2014-03-20 19:39:28,114][INFO ][node ] [Bloodshed]
closing ...
[2014-03-20 19:39:28,119][INFO ][node ] [Bloodshed]
closed

I had changed the config file to

discovery.zen.ping.multicast.enabled: false

and tried restarting the server - no more entries in the log

I also tried moving some of the data (especially that corrupted entry)
from the data directory and restarting. The system message from init.d
would say that ES had started, but then checking the status again it would
stay that it was stopped.

Again no more log entries.

I tried renaming the log file and then touching a file called
elasticsearch.log and chowing it to elasticsearch:elasticsearch | a restart
of the service, it still wasn't running and no log entries

At this point I decided to upgrade my 0.90 ES server to the new 1.0 ES
with a .deb package
I installed, rebooted the machine and after restarting the service - no
entries

My last try was to make a new data directory, a clean data directory, and
then point to that in the config file and restart the ES server.

It still would not start and nothing in the log.

At this point I'm not sure what else to troubleshoot on how to get it back
working again. The data I'm not as worried about, but I need the Elastic
Search server up and running again.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3e438abe-6fed-4333-9155-f6c8e7e2c040%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3e438abe-6fed-4333-9155-f6c8e7e2c040%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM97gdKHm1%3D2V7Gx%2BgZquPa0z_2CMNXudJmoOBbY3DEQ-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3