Exception caught on transport layer

(Koustav Chatterjee) #1

I am using ElasticSearch 2.3 on ec2(ubuntu). I had run sudo apt-get update and upgrade.Then restart the box.

When i start the ES,this is what i run into.

[2017-12-31 11:33:21,605][WARN ][transport.netty          ] [Quasar II] exception caught on transport layer [[id: 0xc3e54945, /0:0:0:0:0:0:0:1:33556 => /0:0:0:0:0:0:0:1:9300]], closing connection
	at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:207)
	at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202)
	at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
	at java.lang.Thread.run(Thread.java:748)

(David Pilato) #2

I removed ---- Please help. It's urgent in the title.

Read this and specifically the "Also be patient" part.

(Koustav Chatterjee) #3


(Jason Tedor) #4

This happens in 2.x if you have an old 1.x node running on the system as well. This is handled more gracefully in later versions.

(Koustav Chatterjee) #5

This is a single node cluster.

(Jason Tedor) #6

That's fine. I'm telling you that I think the old process was still running. The long and the short of this issue is that it's harmless.

(Koustav Chatterjee) #7

The issue was gone after i rebooted the server.

(Jason Tedor) #8

Which would have killed any running processes of an older version. :neutral_face:

(Koustav Chatterjee) #9

Is there a way i can cross check this?
The exception stack thrown does not give enough information.

(Jason Tedor) #10

It happens because of a protocol incompatibility between 1.x and 2.x. The simplest way to reproduce this:

  • start a 1.x node, bind it to localhost
  • start a 2.x node with the defaults

The 2.x node will try to ping the 1.x node. The NPE with exactly the same stacktrace will appear in the logs. Why does this happen? The 1.x node throws an exception because it does not understand the pings from 2.x and tries to send this exception back to the 2.x node. 1.x relied on Java serialization for serializing exceptions. 2.x does not, instead we use our own serialization of exceptions. 2.x sees there is an exception from the remote node, but tries to de-serialize it using our own serialization, rather than the Java serialization that 1.x used. This leads to no exception being read from the wire (in fact, if you run with assertions an assertion will trip instead) and null is returned. This null pointer is then dereferenced, :boom:.

(Koustav Chatterjee) #11

Thanks now i got it.:grinning::grinning::grinning:

(Jason Tedor) #12

You’re very welcome.

(system) closed #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.