# Elasticsearch fails to restart after upgrade

**URL:** <https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821>\
**Category:** Elasticsearch\
**Created:** [December 14, 2017, 5:11pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821 "2017-12-14T17:11:28Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![jattenberg](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jattenberg](https://discuss.elastic.co/u/jattenberg)\
**Post date:** [December 14, 2017, 5:11pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/1 "2017-12-14T17:11:28Z")

</div>

I recently upgraded a local elasticsearch service via apt from 5.2.2 to 5.6.1. Things were working well before, and I've followed the provided upgrade instructions. However, the when attempting to restart the service I see the below error in the logs.

I'm not sure why ES is looking in /var/lib/elasticsearch, I have configured a nonstandard path for the index and logs so that we could use a larger disk. Any help is appreciated.

```auto
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: detected index data in default.path.data [/var/lib/elasticsearch] where there should not be any; check the logs for details
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.5.jar:5.6.5]
Caused by: java.lang.IllegalStateException: detected index data in default.path.data [/var/lib/elasticsearch] where there should not be any; check the logs for details
	at org.elasticsearch.node.Node.checkForIndexDataInDefaultPathData(Node.java:577) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.5.jar:5.6.5]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.5.jar:5.6.5]
	... 6 more

```

---

<div class="post-metadata">

**Author:** ![jattenberg](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jattenberg](https://discuss.elastic.co/u/jattenberg)\
**Post date:** [December 18, 2017, 5:24pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/2 "2017-12-18T17:24:50Z")

</div>

anyone?

---

<div class="post-metadata">

**Author:** ![jattenberg](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jattenberg](https://discuss.elastic.co/u/jattenberg)\
**Post date:** [December 21, 2017, 1:50pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/3 "2017-12-21T13:50:53Z")

</div>

one more try, this is on a client deployment ☹

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [December 21, 2017, 2:03pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/4 "2017-12-21T14:03:25Z")

</div>

You might want to check out [https://github.com/elastic/elasticsearch/pull/24099](https://github.com/elastic/elasticsearch/pull/24099).

You mentioned you use nonstandard paths, what are they?

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [December 21, 2017, 2:48pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/5 "2017-12-21T14:48:14Z")

</div>

As I read the error in your log

ES sees one or more indices in `/var/lib/elasticsearch` where there should not be indices.

Also, `default.path.data` (will not be supported in ESv6) is set to `/var/lib/elasticsearch` somewhere, which is why ES is looking there.

What os are you running @jattenberg? On Debian you should look for `default.path.data` in `/etc/defaults/` or in your ES config dir, probably `/etc/elasticsearch`

What does `ls -l /var/lib/elasticsearch` show?

---

<div class="post-metadata">

**Author:** ![jattenberg](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jattenberg](https://discuss.elastic.co/u/jattenberg)\
**Post date:** [December 21, 2017, 4:30pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/6 "2017-12-21T16:30:56Z")

</div>

@A_B, I'm running ubuntu. The config I have been editing is `/etc/elasticsearch/elasticsearch.yml`, the parameters I've set are:

```auto
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /hd3/elasticsearch/data
#
# Path to log files:
#
path.logs: /hd3/elasticsearch/logs

```

This was working fine prior to the upgrade. hd3 is a new hard drive the client added.

`/var/lib/elasticsearch` has a folder `nodes` with a ton of stuff in it.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [December 21, 2017, 4:40pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/7 "2017-12-21T16:40:31Z")

</div>

> [@jattenberg](#):
>
> /var/lib/elasticsearch has a folder nodes with a ton of stuff in it.

That should not be there I think... That is what ES is complaining about.

Did you grep for `default.path.data` in all of `/etc`?

On Debian I also see some default settings in

> $ grep -r "default.path.data" /etc/\*  
> /etc/init.d/elasticsearch.dpkg-dist:DAEMON\_OPTS="-d -p $PID\_FILE -Edefault.path.logs=$LOG\_DIR -Edefault.path.data=$DATA\_DIR -Edefault.path.conf=$CONF\_DIR"

Which I can see set when I do `ps aux`. Maybe Ubuntu does something similar... I'm running version 5.6.5 with the same settings as version 5.5.2. I have not run version 5.2.2 for a long time so I'm not sure if I have changed something since then...

p.s. depending on what's in the `nodes` folder, I would move it or remove it. I have not needed to move any files and have ES know about them before so I'm not sure what will happen...

---

<div class="post-metadata">

**Author:** ![jattenberg](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jattenberg](https://discuss.elastic.co/u/jattenberg)\
**Post date:** [December 21, 2017, 4:56pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/8 "2017-12-21T16:56:45Z")

</div>

good catch, in the `init.d`, i see: `DAEMON_OPTS="-d -p $PID_FILE -Edefault.path.logs=$LOG_DIR -Edefault.path.data=$DATA_DIR -Edefault.path.conf=$CONF_DIR"` as well.

i revised to `DAEMON_OPTS="-d -p $PID_FILE -Edefault.path.conf=$CONF_DIR"`, but still see the same problem, identical error message when I restart the service. I can delete whatever is in `/var/lib/elasticsearch/nodes`, but I'm concerned ES will still write there.

---

<div class="post-metadata">

**Author:** ![jattenberg](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jattenberg](https://discuss.elastic.co/u/jattenberg)\
**Post date:** [December 21, 2017, 5:00pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/9 "2017-12-21T17:00:13Z")

</div>

same issue if i just change `$DATA_DIR` a `$LOG_DIR` to be what I want them to be. I don't see any other places `default.path.data` is set ☹

---

<div class="post-metadata">

**Author:** ![jattenberg](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jattenberg](https://discuss.elastic.co/u/jattenberg)\
**Post date:** [December 22, 2017, 5:50pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/10 "2017-12-22T17:50:15Z")

</div>

Follow up:

I just tar'd up `/etc/elasticsearch/nodes`, and es started happily. I'm not sure what the contents of that dir were, why they go there, or why es was looking there when I specified a different data path. But, alls well that ends well.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 19, 2018, 5:50pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-restart-after-upgrade/111821/11 "2018-01-19T17:50:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.