Getting ES 6.2.3 error : java.lang.AssertionError: java.text.ParseException: Failed to parse major version from "" (got: )

Hello everyone!

We have a 3 node (1 master only and 2 data nodes) cluster and we are getting the below error intermittently.

The elastic search service fails on the node and after restarting the service, I got the same stack trace again but after a couple of minutes the node gets stable and reallocation starts.

Elastic search version is 6.2.3. Please let me know if you need any more details.

2020-01-03T23:31:43,334][ERROR][o.e.t.n.Netty4Utils ] fatal error on the network layer

        at [org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:184](http://org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:184))

        at [org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:73](http://org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:73))

..
..
.. 1. [2020-01-03T23:31:43,343][WARN ][o.e.t.n.Netty4Transport ] [zdKm3J4] exception caught on transport layer [NettyTcpChannel{localAddress=/172.20.8.20:9300, remoteAddress=/172.20.8.44:45354}], closing connection

  • org.elasticsearch.ElasticsearchException: java.lang.AssertionError: java.text.ParseException: Failed to parse major version from "" (got: )

There doesn't seem to be a network issue during this time.

Complete logs here.

https://pastebin.com/hSw9eNfz

It looks as if some data from another node cannot be read correctly. Are all nodes on the same version?

Also, as 6.2 is somewhat aged, is there any chance you can upgrade to 6.8 and check if that issue persists?

All nodes are in the save version.

I am pushing for an upgrade to a newer version.

One piece of information that can be related is that these ES boxes are running on OpenStack. We have similar configuration on AWS which is stable.

But the boxes on OpenStack have a lot more documents than the one on the AWS.