# Elasticsearch 5.6.3 upgrade to 6.0 Node will not start "failed to read local state"
**URL:** https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252
**Category:** Elasticsearch
**Created:** [November 18, 2017, 10:32pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252 "2017-11-18T22:32:31Z")
**Posts on this page:** 15
**Page:** 2
### Author:  [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [November 24, 2017, 12:27pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/21 "2017-11-24T12:27:18Z")
@tatdat How many nodes do you have in your cluster? Do they all have X-Pack installed and did any of the nodes successfully upgrade before you hit the ML parsing exception?
Is it possible that this node upgraded successfully in which case the cluster state is now version 6.0 and can't be read by version 5.6.2.
The error comes from X-Pack Monitoring but I suspect you would see it even if monitoring was disabled.
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 24, 2017, 4:02pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/22 "2017-11-24T16:02:17Z")
@dkyle. I have 5 node in cluster, 3 data node, 1 client. X-Pack installed in all of them and dont have any node upgrade success to 6.0.0
> [@dkyle](#):
>
> The error comes from X-Pack Monitoring but I suspect you would see it even if monitoring was disabled.
Yeah, before i got this error, i saw some error from x-pack security, but i try to disable x-pack security .
---
### Author:  [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [November 24, 2017, 5:15pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/23 "2017-11-24T17:15:03Z")
Could you paste the full ES log here please
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [November 24, 2017, 5:27pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/24 "2017-11-24T17:27:14Z")
Thanks @dkyle, i sent my log to your private message.
---
### Author:  [@mjett](https://discuss.elastic.co/u/mjett)
#### Post date: [November 27, 2017, 6:19am UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/25 "2017-11-27T06:19:27Z")
I apologize, @dkyle, I should have been more clear.
The APIs do not show any jobs for me to be able to delete, and I don't remember the name to ask for it specifically. As I mentioned, I think the .ml indexes were deleted, so when I query for all I get an index not found.
I am still good while running on 5.6.3, but cannot make the move to 6.0.0 without these errors. It looks like those three types exist somewhere, but I don't have a clue where.
Hope this explains better, and thank you for your help and patience.
---
### Author:  [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [November 27, 2017, 12:55pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/26 "2017-11-27T12:55:35Z")
@mjett
The ML jobs are stored in the Elasticsearch cluster state not the indices. Even after deleting the indices you should be able to list the jobs via the API
```
curl -u 'elastic:changeme' 'http://localhost:9200/_xpack/ml/anomaly_detectors?pretty'
```
To delete a job you must first delete its datafeed
```
curl -X DELETE -u 'elastic:changeme' 'http://localhost:9200/_xpack/ml/datafeeds/your-datafeed'
```
Then the job can be deleted
```
curl -X DELETE -u 'elastic:changeme' 'http://localhost:9200/_xpack/ml/anomaly_detectors/your-job'
```
After this you should be able to upgrade to 6.0
---
### Author:  [@mjett](https://discuss.elastic.co/u/mjett)
#### Post date: [November 27, 2017, 2:32pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/27 "2017-11-27T14:32:59Z")
When I run the first command as listed I don't get any jobs or datafeeds to delete:
```
curl 'http://172.26.10.214:9200/_xpack/ml/anomaly_detectors?pretty'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_expression",
"resource.id" : "_xpack",
"index_uuid" : "_na_",
"index" : "_xpack"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_expression",
"resource.id" : "_xpack",
"index_uuid" : "_na_",
"index" : "_xpack"
},
"status" : 404
}
```
---
### Author:  [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [November 27, 2017, 2:47pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/28 "2017-11-27T14:47:18Z")
It looks like X-Pack is not installed on that node. In your elasticsearch log do you see a message like
`[2017-11-27T12:48:41,705][INFO][o.e.p.PluginsService] [x4qBd0J] loaded plugin [x-pack]`
---
### Author:  [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [November 27, 2017, 3:11pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/29 "2017-11-27T15:11:57Z")
@tatdat Thanks for sending the log file the root cause seems to be a fatal error:
> [2017-11-25T00:19:54,082][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Elastic-03] fatal error in thread [elasticsearch[Elastic-03][clusterService#updateTask][T#1]], exiting
> java.lang.ClassFormatError: Incompatible magic value 1347093252 in class file org/elasticsearch/license/LicenseVerifier
> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0\_121]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0\_121]
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0\_121]
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0\_121]
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0\_121]
For some reason the LicenseVerifier could not be found. This is not my area of expertise I would try creating a new install of elasticsearch and x-pack. If that fails could you create a new question with a more specific title please.
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [November 27, 2017, 4:12pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/30 "2017-11-27T16:12:36Z")
This is interesting: 1347093252 in hex is 0x504B0304. This constant is [famous](https://en.wikipedia.org/wiki/Zip_(file_format)): it's the magic number for ZIP files (the 0x504B is PK for Phil Katz).
However, a JAR file has a different magic constant: [0xCAFEBABE](http://radio-weblogs.com/0100490/2003/01/28.html).
In short, I agree with @dkyle that your installation is messed up and you should reinstall.
---
### Author:  [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [November 27, 2017, 4:39pm UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/31 "2017-11-27T16:39:20Z")
> This constant is famous: it's the magic number for ZIP files (the 0x504B is PK for Phil Katz).
Well I did not know that! I actually decoded the number one byte at a time once I got to PK I assumed it was something to do with keys and was then a little surprised the 3rd byte was end-of-text. Good to know.
---
### Author:  [@mjett](https://discuss.elastic.co/u/mjett)
#### Post date: [November 28, 2017, 5:28am UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/32 "2017-11-28T05:28:50Z")
OK, that question helped immensely @dkyle.
I removed and reinstalled x-pack with bin/elastic-searchplugin on both nodes. I now see the job with the first query.
```
curl -u 'elastic:changeme' 'http://localhost:9200/_xpack/ml/anomaly_detectors?pretty'
```
Once the cluster finishes with shards after restart I'll delete the datafeed and job and try again. Will update with progress.
---
### Author:  [@mjett](https://discuss.elastic.co/u/mjett)
#### Post date: [November 28, 2017, 7:01am UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/33 "2017-11-28T07:01:25Z")
@dkyle: That took care of things, I'm upgrading nodes into 6.0.0 now. Thanks much for the help.
---
### Author:  [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [November 28, 2017, 9:44am UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/34 "2017-11-28T09:44:12Z")
Thanks @mjett that's great news
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 26, 2017, 9:44am UTC](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252/35 "2017-12-26T09:44:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
[Previous page](https://discuss.elastic.co/t/elasticsearch-5-6-3-upgrade-to-6-0-node-will-not-start-failed-to-read-local-state/108252.md?page=1)