Java.lang.IllegalStateException: Message not fully read (request) for requestId

Elastic Stack Elasticsearch
1

I am testing a new ELK cluster.

ES 2.1.1
LS 2.1.1
Kibana 4.3.1

Reading from kafka in LS --> ES. And my ES logs is full of these errors

[2015-12-26 22:33:35,993][WARN ][transport.netty ] [ltx1-app1234] exception caught on transport layer [[id: 0xf4556a30, /10.149.108.27:39248 => /10.149.74.222:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [263646], action [internal:discovery/zen/unicast_gte_1_4], readerIndex [59] vs expected [286]; resetting

It turns out 10.149.108.27 is my tribe node running 1.7.3. So this is just FYI for people who are seeing these errors in their logs. Tribe nodes pre 2.x will not work with ES 2.0 and newer :smile:

2

1.X doesn't work with 2.X, irrespective of what role the node is playing.

3

I may have missed it, but I did not see any mention of this in the 2.x release notes, nor in the breaking changes.

One of the problem is that release notes and documentation for ELK is still bad. It's getting better, but compared to what it used to be :slight_smile: anything is better. I am just trying to share what I learned :smile:

What is needed is a table of the changes between all versions. It seem that users are expected to read all release notes from the version they are at to whichever version it is they are upgrading to.

The release notes only contain the changes between previous and current.

I know it's some work to compile such a table, but once it is done, maintenance is much easier.

Just a suggestion.

4

It's a major release version, and we have never guaranteed compatibility with these.

5

I did not ask for guaranteed compatibility. I am asking for documentation on the changes. Documentation is good. Assuming users will know is bad.

Sorry, I am frustrated. I feel the documentation is lacking. I am spending my holidays working on migration plans from the 1.x series to 2.x and running into issues that is not in the dis-organized (IMHO) documentation.

There need to be a master list of breaking changes for all the various versions that Elastic supports. It need to contain solutions for dealing with the breaking changes.

It would help to have a list of common errors and how to deal with them. I did my due diligence when I encountered these errors, but google-fu did not show anything exactly matching. By having all these common errors reported and indexed by search engines, it makes it easier when others run into the same thing. That was the reason I reported this error. So others can find it and hopefully solve their issue.