Not able to run elasticsearch: elasticsearch dead but subsys locked

[shashwat]

Hi,

I installed elasticsearch 5.5.1. After installation , I was able to run it properly on linux machine. Just, I wanted to access it from my remote windows machine. So, I modified /etc/elasticsearch/elasticsearch.yml file and updated network.host to IP address. Then I restarted the elasticsearch service using
sudo -i service elasticsearch stop
sudo -i service elasticsearch start

but it failed. When I check status of service using:
sudo -i service elasticsearch status , it says:

elasticsearch dead but subsys locked.

Elasticsearch log says:
java.lang.IllegalStateException: detected index data in default.path.data [/var/lib/elasticsearch] where there should not be any; check the logs for details.

Thanks,

[talevy (Tal Levy)]

Hi @shashwat,

I believe you are running into this: https://github.com/elastic/elasticsearch/pull/24099

We made it so we double-check whether default.path.data & path.data are set and
hope to verify that we are not writing shard data to both.

Although the network.host setting may have seemed like the culprit, I believe it is how
you are setting path.data that is the problem. Did you run Elasticsearch with default settings, index data, and then change path.data by chance?

Steps for how I reproduce this problem to illustrate using a fresh install of Elasticsearch

1. run Elasticsearch with the default default.data.path, in the package's case it is /var/lib/elasticsearch

bin/elasticsearch -Edefault.path.data=/tmp/data

2. index data

curl -XPUT 'localhost:9200/twitter/tweet/1' -H 'Content-Type: application/json' -d'                                         
{                                
    "user" : "kimchy",           
    "post_date" : "2009-11-15T14:12:12",                          
    "message" : "trying out Elasticsearch"                        
}                                
'

3. Run Elasticsearch with a data.path that is different to default.data.path, but explicitely set the default path (once again, this is being done for you in the packaging)

bin/elasticsearch -Edefault.path.data=/tmp/data -Epath.data=/tmp/different_path
...
[2017-08-10T11:20:30,607][INFO ][o.e.n.Node               ] index folder [6td_4mpERZOtJ8DvVyPU4w] in default.path.data [/tmp/data/no$es/0/indices] must be moved to any of [/tmp/different_path/nodes/0/indices] 
...
[2017-08-10T11:20:30,614][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]                 
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: detected index data in default.path.data [/tmp/data] $here there should not be any; check the logs for details
...

Double check you are writing your data to the correct directory