"Message not fully read (response)" after upgrading 0.90.0 to 0.90.1

Dave_Konopka · July 19, 2013, 3:51pm

We recently updated a three node cluster from 0.90.0 to 0.90.1. Now one of
the nodes is intermittently showing messages like these in the logs:

[transport.netty] Message not fully read (response) for [12652] handler org.
elasticsearch.action.support.master.
TransportMasterNodeOperationAction$4@53854b5f, error [true], resetting

I saw an earlier thread that this could be an issue with 0.90.0 that was
fixed in 0.90.1. Odd though that it seems to have started for us with 0.90.1

Anyone have any thoughts or suggestions?

The nodes are all running java version "1.6.0_27" openjdk.

{
"ok" : true,
"status" : 200,
"name" : "...",
"version" : {
"number" : "0.90.1",
"snapshot_build" : false,
"lucene_version" : "4.3"
},
"tagline" : "You Know, for Search"
}

Cluster health:
{
"cluster_name" : "staging_cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 9,
"active_shards" : 19,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

brian_yoder · July 19, 2013, 4:35pm

Oracle (formerly Sun) Java is good. Java 6 works for me (political reasons
keep me from going to Java 7 at the moment), though java 7 is recommended.

OpenJDK is not good. It's "exactly the same as Oracle Java, except that
it's different". And it's those little differences that will cripple your
application with strange and unpredictable failures.

On Friday, July 19, 2013 11:51:23 AM UTC-4, Dave Konopka wrote:

We recently updated a three node cluster from 0.90.0 to 0.90.1. Now one of
the nodes is intermittently showing messages like these in the logs:

[transport.netty] Message not fully read (response) for [12652] handler
org.elasticsearch.action.support.master.
TransportMasterNodeOperationAction$4@53854b5f, error [true], resetting

I saw an earlier thread that this could be an issue with 0.90.0 that was
fixed in 0.90.1. Odd though that it seems to have started for us with 0.90.1

Anyone have any thoughts or suggestions?

The nodes are all running java version "1.6.0_27" openjdk.

{
"ok" : true,
"status" : 200,
"name" : "...",
"version" : {
"number" : "0.90.1",
"snapshot_build" : false,
"lucene_version" : "4.3"
},
"tagline" : "You Know, for Search"
}

Cluster health:
{
"cluster_name" : "staging_cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 9,
"active_shards" : 19,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Dave_Konopka · July 19, 2013, 7:45pm

We've been running OpenJDK for a while without issues. This only started
happening after the upgrade to 0.90.1.

Is Oracle Java recommended officially over OpenJDK?

On Fri, Jul 19, 2013 at 12:35 PM, InquiringMind brian.from.fl@gmail.comwrote:

Oracle (formerly Sun) Java is good. Java 6 works for me (political reasons
keep me from going to Java 7 at the moment), though java 7 is recommended.

OpenJDK is not good. It's "exactly the same as Oracle Java, except that
it's different". And it's those little differences that will cripple your
application with strange and unpredictable failures.

On Friday, July 19, 2013 11:51:23 AM UTC-4, Dave Konopka wrote:

We recently updated a three node cluster from 0.90.0 to 0.90.1. Now one
of the nodes is intermittently showing messages like these in the logs:

[transport.netty] Message not fully read (response) for [12652] handler
org.elasticsearch.action.support.master.TransportMasterNodeO
perationAction$4@53854b5f, error [true], resetting

I saw an earlier thread that this could be an issue with 0.90.0 that was
fixed in 0.90.1. Odd though that it seems to have started for us with 0.90.1

Anyone have any thoughts or suggestions?

The nodes are all running java version "1.6.0_27" openjdk.

{
"ok" : true,
"status" : 200,
"name" : "...",
"version" : {
"number" : "0.90.1",
"snapshot_build" : false,
"lucene_version" : "4.3"
},
"tagline" : "You Know, for Search"
}

Cluster health:
{
"cluster_name" : "staging_cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 9,
"active_shards" : 19,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

brian_yoder · July 19, 2013, 8:08pm

Well http://elasticsearch-users.115913.n3.nabble.com/Best-java-to-run-ElasticSearch-td4030260.html
indicates you should use the one that works!

Personally, I am currently on ES 0.90.0 (not .1) and on Oracle Java 6
(latest versions for Mac, Linux, and Solaris x86). But that's anecdotal
evidence, not scientific!

Brian

On Friday, July 19, 2013 3:45:27 PM UTC-4, Dave Konopka wrote:

We've been running OpenJDK for a while without issues. This only started
happening after the upgrade to 0.90.1.

Is Oracle Java recommended officially over OpenJDK?

On Fri, Jul 19, 2013 at 12:35 PM, InquiringMind <brian....@gmail.com<javascript:>

wrote:

Oracle (formerly Sun) Java is good. Java 6 works for me (political
reasons keep me from going to Java 7 at the moment), though java 7 is
recommended.

OpenJDK is not good. It's "exactly the same as Oracle Java, except that
it's different". And it's those little differences that will cripple your
application with strange and unpredictable failures.

On Friday, July 19, 2013 11:51:23 AM UTC-4, Dave Konopka wrote:

We recently updated a three node cluster from 0.90.0 to 0.90.1. Now one
of the nodes is intermittently showing messages like these in the logs:

[transport.netty] Message not fully read (response) for [12652] handler
org.elasticsearch.action.support.master.TransportMasterNodeO
perationAction$4@53854b5f, error [true], resetting

I saw an earlier thread that this could be an issue with 0.90.0 that was
fixed in 0.90.1. Odd though that it seems to have started for us with 0.90.1

Anyone have any thoughts or suggestions?

The nodes are all running java version "1.6.0_27" openjdk.

{
"ok" : true,
"status" : 200,
"name" : "...",
"version" : {
"number" : "0.90.1",
"snapshot_build" : false,
"lucene_version" : "4.3"
},
"tagline" : "You Know, for Search"
}

Cluster health:
{
"cluster_name" : "staging_cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 9,
"active_shards" : 19,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

drewr · July 19, 2013, 8:46pm

Dave Konopka wrote:

We recently updated a three node cluster from 0.90.0 to
0.90.1. Now one of the nodes is intermittently showing messages
like these in the logs:

[transport.netty] Message not fully read (response) for [12652]
handler org.
elasticsearch.action.support.master.
TransportMasterNodeOperationAction$4@53854b5f, error [true],
resetting

You have some node that's likely still on 0.90.0. Try running
this on one of your nodes and make sure each line is the same.

curl -s users.elasticsearch.org/drewr/es-version.sh | sh

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Dave_Konopka · July 22, 2013, 12:29pm

Hi Drew,

All nodes are reporting 0.9.1 and the same Java version:
0.90.1 OpenJDK 64-Bit Server VM 1.6.0_27 inet[/0.0.0.0:9300]
0.90.1 OpenJDK 64-Bit Server VM 1.6.0_27 inet[/0.0.0.0:9300]
0.90.1 OpenJDK 64-Bit Server VM 1.6.0_27 inet[/0.0.0.0:9300]

Dave

On Fri, Jul 19, 2013 at 4:46 PM, Drew Raines aaraines@gmail.com wrote:

Dave Konopka wrote:

We recently updated a three node cluster from 0.90.0 to 0.90.1. Now one

of the nodes is intermittently showing messages like these in the logs:

[transport.netty] Message not fully read (response) for [12652] handler
org.
elasticsearch.action.support.master.
TransportMasterNodeOperationAction$4@53854b5f, error [true], resetting

You have some node that's likely still on 0.90.0. Try running this on one
of your nodes and make sure each line is the same.

curl -s users.elasticsearch.org/drewr/**es-version.sh http://users.elasticsearch.org/drewr/es-version.sh| sh

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Dave_Konopka · July 22, 2013, 3:52pm

java -version revealed that IcedTea6 version was different on one node. I
updated that node (which rebuilt the jre libs) and things seem to be
running smoothly again.

On Mon, Jul 22, 2013 at 8:29 AM, Dave Konopka dave.konopka@gmail.comwrote:

Hi Drew,

All nodes are reporting 0.9.1 and the same Java version:
0.90.1 OpenJDK 64-Bit Server VM 1.6.0_27 inet[/0.0.0.0:9300]
0.90.1 OpenJDK 64-Bit Server VM 1.6.0_27 inet[/0.0.0.0:9300]
0.90.1 OpenJDK 64-Bit Server VM 1.6.0_27 inet[/0.0.0.0:9300]

Dave

On Fri, Jul 19, 2013 at 4:46 PM, Drew Raines aaraines@gmail.com wrote:

Dave Konopka wrote:

We recently updated a three node cluster from 0.90.0 to 0.90.1. Now one

of the nodes is intermittently showing messages like these in the logs:

[transport.netty] Message not fully read (response) for [12652] handler
org.
elasticsearch.action.support.master.
TransportMasterNodeOperationAction$4@53854b5f, error [true], resetting

You have some node that's likely still on 0.90.0. Try running this on
one of your nodes and make sure each line is the same.

curl -s users.elasticsearch.org/drewr/**es-version.sh http://users.elasticsearch.org/drewr/es-version.sh| sh

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.