Failing starting due to transport layer exception

lcxlcx · April 15, 2013, 3:15am

Version : 0.20.5

I have installed ES as usual, everything works well except
that it issue an exception during starting, but it still could
work.

log snippt :

exception caught on transport layer [[id: 0x6d33e262,
/192.168.2.183:58596 :> /192.168.13.89:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format
at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:482)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:336)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:570)
at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:812)
at org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
at org.elasticsearch.transport.netty.NettyTransport.exceptionCaught(NettyTransport.java:500)
at org.elasticsearch.transport.netty.MessageChannelHandler.exceptionCaught(MessageChannelHandler.java:227)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:654)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

It just issues this exception sevral times, but it still starts
successfully eventually.

I have googled it, found this may be caused by the version mismatch of
ES and logstash, but I don't use logstash.

--

Regards

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

spinscale · April 15, 2013, 6:21am

Hey,

what are you doing to trigger this exception? Are you connecting via Java
to your elasticsearch instance? If so, maybe you are you using two
different versions of elasticsearch on server and client side. If not,
please tell, what you are doing.

--Alex

On Mon, Apr 15, 2013 at 5:15 AM, larmbr zhan nasa4836@gmail.com wrote:

Version : 0.20.5

I have installed ES as usual, everything works well except
that it issue an exception during starting, but it still could
work.

log snippt :

exception caught on transport layer [[id: 0x6d33e262,
/192.168.2.183:58596 :> /192.168.13.89:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:482)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:336)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81)
at
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:570)
at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:812)
at
org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
at
org.elasticsearch.transport.netty.NettyTransport.exceptionCaught(NettyTransport.java:500)
at
org.elasticsearch.transport.netty.MessageChannelHandler.exceptionCaught(MessageChannelHandler.java:227)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at
org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:654)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

It just issues this exception sevral times, but it still starts
successfully eventually.

I have googled it, found this may be caused by the version mismatch of
ES and logstash, but I don't use logstash.

--

Regards

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

lcxlcx · April 15, 2013, 6:29am

On Mon, Apr 15, 2013 at 2:21 PM, Alexander Reelsen alr@spinscale.de wrote:

please tell, what you are doing.

Hi,

I just start an elasticsearch instance, and there are only one version
of ES(0.20.5)
and only one instance running.

--

Regards,
Zhan Jianyu

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

spinscale · April 15, 2013, 7:03am

hey,

the exception is showing, that a network connection is being opened, so
have to do something (trying to connect to elasticsearch) - the line
192.168.2.183:58596 :> /192.168.13.89:9300 tells that.

Are you maybe using HTTP to connect to port 9300 instead of 9200?

--Alex

On Mon, Apr 15, 2013 at 8:29 AM, larmbr zhan nasa4836@gmail.com wrote:

On Mon, Apr 15, 2013 at 2:21 PM, Alexander Reelsen alr@spinscale.de
wrote:

please tell, what you are doing.

Hi,

I just start an elasticsearch instance, and there are only one version
of ES(0.20.5)
and only one instance running.

--

Regards,
Zhan Jianyu

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

lcxlcx · April 15, 2013, 7:35am

On Mon, Apr 15, 2013 at 3:03 PM, Alexander Reelsen alr@spinscale.de wrote:

Are you maybe using HTTP to connect to port 9300 instead of 9200?

Hi, I konw this difference.
Via command line , I use HTTP to connect to 9200,
Via Java API, I use TCP to connect to 9300.

And exactly this exception happens during the starting of
elasticsearch, during which
I launch no requests to ES.

And as I menstioned above, it finally starts successfully and has no
impact on my later usage.
I just thought it is an eyesore and want to get rid of it.

--

Regards,
Zhan Jianyu

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · April 15, 2013, 8:02am

May be you have another ES process running on you network?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 15 avr. 2013 à 09:35, larmbr zhan nasa4836@gmail.com a écrit :

On Mon, Apr 15, 2013 at 3:03 PM, Alexander Reelsen alr@spinscale.de wrote:

Are you maybe using HTTP to connect to port 9300 instead of 9200?

Hi, I konw this difference.
Via command line , I use HTTP to connect to 9200,
Via Java API, I use TCP to connect to 9300.

And exactly this exception happens during the starting of
elasticsearch, during which
I launch no requests to ES.

And as I menstioned above, it finally starts successfully and has no
impact on my later usage.
I just thought it is an eyesore and want to get rid of it.

--

Regards,
Zhan Jianyu

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

spinscale · April 15, 2013, 8:05am

Hey

ok, then its best to just ignore this, because there is another host
connecting to your elasticsearch instance, but obviously not sending the
data needed for correct communication. There is not much more to do to able
to debug this outside of your network. You might want to find out, who is
actually connecting and why (I am not aware of anything known running on
port 9300), then you could actually disable it and make the exception go
away. Another simple workaround is to start on a different port
(setting transport.tcp.port) and see if the node still connects. I wouldnt
spent too much time with this...

On Mon, Apr 15, 2013 at 9:35 AM, larmbr zhan nasa4836@gmail.com wrote:

On Mon, Apr 15, 2013 at 3:03 PM, Alexander Reelsen alr@spinscale.de
wrote:

Are you maybe using HTTP to connect to port 9300 instead of 9200?

Hi, I konw this difference.
Via command line , I use HTTP to connect to 9200,
Via Java API, I use TCP to connect to 9300.

And exactly this exception happens during the starting of
elasticsearch, during which
I launch no requests to ES.

And as I menstioned above, it finally starts successfully and has no
impact on my later usage.
I just thought it is an eyesore and want to get rid of it.

--

Regards,
Zhan Jianyu

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dk1 · April 16, 2013, 1:06pm

Hi

In my case, I see

[2013-04-16 12:56:42,012][INFO ][node ] [Norns] {0.20.6}[9171]:
initializing ...
[2013-04-16 12:56:42,045][INFO ][plugins ] [Norns] loaded , sites

[2013-04-16 12:56:49,557][INFO ][node ] [Norns] {0.20.6}[9171]:
initialized
[2013-04-16 12:56:49,557][INFO ][node ] [Norns] {0.20.6}[9171]:
starting ...
[2013-04-16 12:56:49,845][INFO ][transport ] [Norns] bound_address
{inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
[2013-04-16 12:56:52,938][INFO ][cluster.service ] [Norns] new_master
[Norns][Ump8lBn3Rv6SP-DX0Un4Rw][inet[/127.0.0.1:9300]], reason:
zen-disco-join (elected_as_master)
[2013-04-16 12:56:53,094][INFO ][discovery ] [Norns]
test_es_cluster/Ump8lBn3Rv6SP-DX0Un4Rw
[2013-04-16 12:56:53,226][INFO ][http ] [Norns] bound_address
{inet[/127.0.0.1:9200]}, publish_address {inet[/127.0.0.1:9200]}
[2013-04-16 12:56:53,226][INFO ][node ] [Norns] {0.20.6}[9171]:
started
[2013-04-16 12:56:53,281][INFO ][gateway ] [Norns] recovered [0]
indices into cluster_state
[2013-04-16 12:56:53,527][WARN ][transport.netty ] [Norns] exception caught
on transport layer [[id: 0xfeb8c270, /127.0.0.1:47797 => /127.0.0.1:9300]],
closing connection
java.io.StreamCorruptedException: invalid internal transport message format

All other symptoms are the same.
ES 0.20.6 installed on AWS instance (multicast disabled by default);
elasticsearch.yml contains

network.host: 127.0.0.1

I was trying open all necessary ports for ES on instance - nothing changed.

Denis

On Monday, April 15, 2013 11:05:22 AM UTC+3, Alexander Reelsen wrote:

Hey

ok, then its best to just ignore this, because there is another host
connecting to your elasticsearch instance, but obviously not sending the
data needed for correct communication. There is not much more to do to able
to debug this outside of your network. You might want to find out, who is
actually connecting and why (I am not aware of anything known running on
port 9300), then you could actually disable it and make the exception go
away. Another simple workaround is to start on a different port
(setting transport.tcp.port) and see if the node still connects. I wouldnt
spent too much time with this...

On Mon, Apr 15, 2013 at 9:35 AM, larmbr zhan <nasa...@gmail.com<javascript:>

wrote:

On Mon, Apr 15, 2013 at 3:03 PM, Alexander Reelsen <a...@spinscale.de<javascript:>>
wrote:

Are you maybe using HTTP to connect to port 9300 instead of 9200?

Hi, I konw this difference.
Via command line , I use HTTP to connect to 9200,
Via Java API, I use TCP to connect to 9300.

And exactly this exception happens during the starting of
elasticsearch, during which
I launch no requests to ES.

And as I menstioned above, it finally starts successfully and has no
impact on my later usage.
I just thought it is an eyesore and want to get rid of it.

--

Regards,
Zhan Jianyu

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

lcxlcx · April 16, 2013, 1:39pm

On Mon, Apr 15, 2013 at 4:05 PM, Alexander Reelsen alr@spinscale.de wrote:

ok, then its best to just ignore this, because there is another host
connecting to your elasticsearch instance, but obviously not sending the
data needed for correct communication. There is not much more to do to able
to debug this outside of your network. You might want to find out, who is
actually connecting and why (I am not aware of anything known running on
port 9300), then you could actually disable it and make the exception go
away. Another simple workaround is to start on a different port (setting
transport.tcp.port) and see if the node still connects. I wouldnt spent too
much time with this...

yep, I think there is some one in my local network using ES too. And he/she
happens to use the same port . Thanks for your advice.

--

Regards,
Zhan Jianyu

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.