Elasticsearch fails to restart after upgrade

Elastic Stack Elasticsearch
1

I recently upgraded a local elasticsearch service via apt from 5.2.2 to 5.6.1. Things were working well before, and I've followed the provided upgrade instructions. However, the when attempting to restart the service I see the below error in the logs.

I'm not sure why ES is looking in /var/lib/elasticsearch, I have configured a nonstandard path for the index and logs so that we could use a larger disk. Any help is appreciated.

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: detected index data in default.path.data [/var/lib/elasticsearch] where there should not be any; check the logs for details
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.5.jar:5.6.5]
Caused by: java.lang.IllegalStateException: detected index data in default.path.data [/var/lib/elasticsearch] where there should not be any; check the logs for details
	at org.elasticsearch.node.Node.checkForIndexDataInDefaultPathData(Node.java:577) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.5.jar:5.6.5]
	... 6 more
2

anyone?

3

one more try, this is on a client deployment :frowning:

4

You might want to check out https://github.com/elastic/elasticsearch/pull/24099.

You mentioned you use nonstandard paths, what are they?

5

As I read the error in your log

ES sees one or more indices in /var/lib/elasticsearch where there should not be indices.

Also, default.path.data (will not be supported in ESv6) is set to /var/lib/elasticsearch somewhere, which is why ES is looking there.

What os are you running @jattenberg? On Debian you should look for default.path.data in /etc/defaults/ or in your ES config dir, probably /etc/elasticsearch

What does ls -l /var/lib/elasticsearch show?

6

@A_B, I'm running ubuntu. The config I have been editing is /etc/elasticsearch/elasticsearch.yml, the parameters I've set are:

# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /hd3/elasticsearch/data
#
# Path to log files:
#
path.logs: /hd3/elasticsearch/logs

This was working fine prior to the upgrade. hd3 is a new hard drive the client added.

/var/lib/elasticsearch has a folder nodes with a ton of stuff in it.

7

That should not be there I think... That is what ES is complaining about.

Did you grep for default.path.data in all of /etc?

On Debian I also see some default settings in

$ grep -r "default.path.data" /etc/*
/etc/init.d/elasticsearch.dpkg-dist:DAEMON_OPTS="-d -p $PID_FILE -Edefault.path.logs=$LOG_DIR -Edefault.path.data=$DATA_DIR -Edefault.path.conf=$CONF_DIR"

Which I can see set when I do ps aux. Maybe Ubuntu does something similar... I'm running version 5.6.5 with the same settings as version 5.5.2. I have not run version 5.2.2 for a long time so I'm not sure if I have changed something since then...

p.s. depending on what's in the nodes folder, I would move it or remove it. I have not needed to move any files and have ES know about them before so I'm not sure what will happen...

8

good catch, in the init.d, i see: DAEMON_OPTS="-d -p $PID_FILE -Edefault.path.logs=$LOG_DIR -Edefault.path.data=$DATA_DIR -Edefault.path.conf=$CONF_DIR" as well.

i revised to DAEMON_OPTS="-d -p $PID_FILE -Edefault.path.conf=$CONF_DIR", but still see the same problem, identical error message when I restart the service. I can delete whatever is in /var/lib/elasticsearch/nodes, but I'm concerned ES will still write there.

9

same issue if i just change $DATA_DIR a $LOG_DIR to be what I want them to be. I don't see any other places default.path.data is set :frowning:

10

Follow up:

I just tar'd up /etc/elasticsearch/nodes, and es started happily. I'm not sure what the contents of that dir were, why they go there, or why es was looking there when I specified a different data path. But, alls well that ends well.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.