Exception caught on transport layer

Elastic Stack Elasticsearch
1

I am using ElasticSearch 2.3 on ec2(ubuntu). I had run sudo apt-get update and upgrade.Then restart the box.

When i start the ES,this is what i run into.

[2017-12-31 11:33:21,605][WARN ][transport.netty          ] [Quasar II] exception caught on transport layer [[id: 0xc3e54945, /0:0:0:0:0:0:0:1:33556 => /0:0:0:0:0:0:0:1:9300]], closing connection
java.lang.NullPointerException
	at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:207)
	at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202)
	at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
	at java.lang.Thread.run(Thread.java:748)
2

I removed ---- Please help. It's urgent in the title.

Read this and specifically the "Also be patient" part.

3

Thanks.

4

This happens in 2.x if you have an old 1.x node running on the system as well. This is handled more gracefully in later versions.

5

This is a single node cluster.

6

That's fine. I'm telling you that I think the old process was still running. The long and the short of this issue is that it's harmless.

7

The issue was gone after i rebooted the server.

8

Which would have killed any running processes of an older version. :neutral_face:

9

Is there a way i can cross check this?
The exception stack thrown does not give enough information.

10

It happens because of a protocol incompatibility between 1.x and 2.x. The simplest way to reproduce this:

  • start a 1.x node, bind it to localhost
  • start a 2.x node with the defaults

The 2.x node will try to ping the 1.x node. The NPE with exactly the same stacktrace will appear in the logs. Why does this happen? The 1.x node throws an exception because it does not understand the pings from 2.x and tries to send this exception back to the 2.x node. 1.x relied on Java serialization for serializing exceptions. 2.x does not, instead we use our own serialization of exceptions. 2.x sees there is an exception from the remote node, but tries to de-serialize it using our own serialization, rather than the Java serialization that 1.x used. This leads to no exception being read from the wire (in fact, if you run with assertions an assertion will trip instead) and null is returned. This null pointer is then dereferenced, :boom:.

11

Thanks now i got it.:grinning::grinning::grinning:

12

You’re very welcome.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.