Elasticsearch 5.6.3 upgrade to 6.0 Node will not start "failed to read local state"

Elastic Stack Elasticsearch
21

@tatdat How many nodes do you have in your cluster? Do they all have X-Pack installed and did any of the nodes successfully upgrade before you hit the ML parsing exception?

Is it possible that this node upgraded successfully in which case the cluster state is now version 6.0 and can't be read by version 5.6.2.

The error comes from X-Pack Monitoring but I suspect you would see it even if monitoring was disabled.

22

@dkyle. I have 5 node in cluster, 3 data node, 1 client. X-Pack installed in all of them and dont have any node upgrade success to 6.0.0

Yeah, before i got this error, i saw some error from x-pack security, but i try to disable x-pack security .

23

Could you paste the full ES log here please

24

Thanks @dkyle, i sent my log to your private message.

25

I apologize, @dkyle, I should have been more clear.

The APIs do not show any jobs for me to be able to delete, and I don't remember the name to ask for it specifically. As I mentioned, I think the .ml indexes were deleted, so when I query for all I get an index not found.

I am still good while running on 5.6.3, but cannot make the move to 6.0.0 without these errors. It looks like those three types exist somewhere, but I don't have a clue where.

Hope this explains better, and thank you for your help and patience.

26

@mjett

The ML jobs are stored in the Elasticsearch cluster state not the indices. Even after deleting the indices you should be able to list the jobs via the API

curl -u 'elastic:changeme'  'http://localhost:9200/_xpack/ml/anomaly_detectors?pretty'

To delete a job you must first delete its datafeed

curl -X DELETE -u 'elastic:changeme' 'http://localhost:9200/_xpack/ml/datafeeds/your-datafeed'

Then the job can be deleted

curl -X DELETE -u 'elastic:changeme' 'http://localhost:9200/_xpack/ml/anomaly_detectors/your-job'

After this you should be able to upgrade to 6.0

27

When I run the first command as listed I don't get any jobs or datafeeds to delete:

curl 'http://172.26.10.214:9200/_xpack/ml/anomaly_detectors?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "resource.type" : "index_expression",
        "resource.id" : "_xpack",
        "index_uuid" : "_na_",
        "index" : "_xpack"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "resource.type" : "index_expression",
    "resource.id" : "_xpack",
    "index_uuid" : "_na_",
    "index" : "_xpack"
  },
  "status" : 404
}
28

It looks like X-Pack is not installed on that node. In your elasticsearch log do you see a message like

[2017-11-27T12:48:41,705][INFO ][o.e.p.PluginsService ] [x4qBd0J] loaded plugin [x-pack]

29

@tatdat Thanks for sending the log file the root cause seems to be a fatal error:

[2017-11-25T00:19:54,082][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [Elastic-03] fatal error in thread [elasticsearch[Elastic-03][clusterService#updateTask][T#1]], exiting
java.lang.ClassFormatError: Incompatible magic value 1347093252 in class file org/elasticsearch/license/LicenseVerifier
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_121]
at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_121]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_121]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0_121]
at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_121]

For some reason the LicenseVerifier could not be found. This is not my area of expertise I would try creating a new install of elasticsearch and x-pack. If that fails could you create a new question with a more specific title please.

30

This is interesting: 1347093252 in hex is 0x504B0304. This constant is famous: it's the magic number for ZIP files (the 0x504B is PK for Phil Katz).

However, a JAR file has a different magic constant: 0xCAFEBABE.

In short, I agree with @dkyle that your installation is messed up and you should reinstall.

1 Like
31

This constant is famous: it's the magic number for ZIP files (the 0x504B is PK for Phil Katz).

Well I did not know that! I actually decoded the number one byte at a time once I got to PK I assumed it was something to do with keys and was then a little surprised the 3rd byte was end-of-text. Good to know.

32

OK, that question helped immensely @dkyle.

I removed and reinstalled x-pack with bin/elastic-searchplugin on both nodes. I now see the job with the first query.

curl -u 'elastic:changeme'  'http://localhost:9200/_xpack/ml/anomaly_detectors?pretty'

Once the cluster finishes with shards after restart I'll delete the datafeed and job and try again. Will update with progress.

33

@dkyle: That took care of things, I'm upgrading nodes into 6.0.0 now. Thanks much for the help.

34

Thanks @mjett that's great news

35

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.