Exception caught on transport layer (0.20.1)

Elastic Stack Elasticsearch
1

I run elasticsearch on a debian server. It starts up, the log is fine until
I run logstash. Then I start getting errors in the elasticsearch logs as
below. Any idea what is going on?

Thanks!
Ognen

[2012-12-18 10:51:42,111][INFO ][node ] [command
center] {0.20.1}[8901]: initializing ...
[2012-12-18 10:51:42,122][INFO ][plugins ] [command
center] loaded [], sites []
[2012-12-18 10:51:46,686][INFO ][node ] [command
center] {0.20.1}[8901]: initialized
[2012-12-18 10:51:46,686][INFO ][node ] [command
center] {0.20.1}[8901]: starting ...
[2012-12-18 10:51:46,879][INFO ][transport ] [command
center] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.6.0.88:9300]}
[2012-12-18 10:51:49,933][INFO ][cluster.service ] [command
center] new_master [command
center][erqnAKNCQUu_tUyLUSgS6w][inet[/10.6.0.88:9300]]{master=true},
reason: zen-disco-join (elected_as_master)
[2012-12-18 10:51:49,992][INFO ][discovery ] [command
center] [server]/erqnAKNCQUu_tUyLUSgS6w
[2012-12-18 10:51:50,042][INFO ][http ] [command
center] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.6.0.88:9200]}
[2012-12-18 10:51:50,043][INFO ][node ] [command
center] {0.20.1}[8901]: started
[2012-12-18 10:51:50,230][INFO ][gateway ] [command
center] recovered [0] indices into cluster_state
[2012-12-18 10:54:31,674][WARN ][transport.netty ] [command
center] exception caught on transport layer [[id: 0x4d0e98fc,
/10.6.0.88:58532 => /10.6.0.88:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
enthought-es.log

--

2

Are all other nodes also running version 0.20.1?

On 18 December 2012 18:02, Ognen Duzlevski ognen@enthought.com wrote:

I run elasticsearch on a debian server. It starts up, the log is fine until
I run logstash. Then I start getting errors in the elasticsearch logs as
below. Any idea what is going on?

Thanks!
Ognen

[2012-12-18 10:51:42,111][INFO ][node ] [command center]
{0.20.1}[8901]: initializing ...
[2012-12-18 10:51:42,122][INFO ][plugins ] [command center]
loaded , sites
[2012-12-18 10:51:46,686][INFO ][node ] [command center]
{0.20.1}[8901]: initialized
[2012-12-18 10:51:46,686][INFO ][node ] [command center]
{0.20.1}[8901]: starting ...
[2012-12-18 10:51:46,879][INFO ][transport ] [command center]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.6.0.88:9300]}
[2012-12-18 10:51:49,933][INFO ][cluster.service ] [command center]
new_master [command
center][erqnAKNCQUu_tUyLUSgS6w][inet[/10.6.0.88:9300]]{master=true}, reason:
zen-disco-join (elected_as_master)
[2012-12-18 10:51:49,992][INFO ][discovery ] [command center]
[server]/erqnAKNCQUu_tUyLUSgS6w
[2012-12-18 10:51:50,042][INFO ][http ] [command center]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.6.0.88:9200]}
[2012-12-18 10:51:50,043][INFO ][node ] [command center]
{0.20.1}[8901]: started
[2012-12-18 10:51:50,230][INFO ][gateway ] [command center]
recovered [0] indices into cluster_state
[2012-12-18 10:54:31,674][WARN ][transport.netty ] [command center]
exception caught on transport layer [[id: 0x4d0e98fc, /10.6.0.88:58532 =>
/10.6.0.88:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
enthought-es.log

--

--
Met vriendelijke groet,

Martijn van Groningen

--

3

On Tuesday, December 18, 2012 2:05:24 PM UTC-6, Martijn v Groningen wrote:

Are all other nodes also running version 0.20.1?

No other nodes.
OD

On 18 December 2012 18:02, Ognen Duzlevski <og...@enthought.com<javascript:>>
wrote:

I run elasticsearch on a debian server. It starts up, the log is fine
until
I run logstash. Then I start getting errors in the elasticsearch logs as
below. Any idea what is going on?

Thanks!
Ognen

[2012-12-18 10:51:42,111][INFO ][node ] [command
center]
{0.20.1}[8901]: initializing ...
[2012-12-18 10:51:42,122][INFO ][plugins ] [command
center]
loaded , sites
[2012-12-18 10:51:46,686][INFO ][node ] [command
center]
{0.20.1}[8901]: initialized
[2012-12-18 10:51:46,686][INFO ][node ] [command
center]
{0.20.1}[8901]: starting ...
[2012-12-18 10:51:46,879][INFO ][transport ] [command
center]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.6.0.88:9300]}
[2012-12-18 10:51:49,933][INFO ][cluster.service ] [command
center]
new_master [command
center][erqnAKNCQUu_tUyLUSgS6w][inet[/10.6.0.88:9300]]{master=true},
reason:
zen-disco-join (elected_as_master)
[2012-12-18 10:51:49,992][INFO ][discovery ] [command
center]
[server]/erqnAKNCQUu_tUyLUSgS6w
[2012-12-18 10:51:50,042][INFO ][http ] [command
center]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.6.0.88:9200]}
[2012-12-18 10:51:50,043][INFO ][node ] [command
center]
{0.20.1}[8901]: started
[2012-12-18 10:51:50,230][INFO ][gateway ] [command
center]
recovered [0] indices into cluster_state
[2012-12-18 10:54:31,674][WARN ][transport.netty ] [command
center]
exception caught on transport layer [[id: 0x4d0e98fc, /10.6.0.88:58532=>
/10.6.0.88:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message
format
at

org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)

    at

org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)

    at

org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)

    at

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786)

    at

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)

    at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)

    at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)

    at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)

    at

org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)

    at

org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)

    at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)

    at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)

enthought-es.log

--

--
Met vriendelijke groet,

Martijn van Groningen

--

4

Sorry, I meant to say, I am not running any other nodes, this is just an
experimental setup to see if I can ship syslogs from various machines to
logsash and use a standalone elasticsearch.
Ognen

On Tuesday, December 18, 2012 10:21:01 PM UTC-6, Ognen Duzlevski wrote:

On Tuesday, December 18, 2012 2:05:24 PM UTC-6, Martijn v Groningen wrote:

Are all other nodes also running version 0.20.1?

No other nodes.
OD

On 18 December 2012 18:02, Ognen Duzlevski og...@enthought.com wrote:

I run elasticsearch on a debian server. It starts up, the log is fine
until
I run logstash. Then I start getting errors in the elasticsearch logs
as
below. Any idea what is going on?

Thanks!
Ognen

[2012-12-18 10:51:42,111][INFO ][node ] [command
center]
{0.20.1}[8901]: initializing ...
[2012-12-18 10:51:42,122][INFO ][plugins ] [command
center]
loaded , sites
[2012-12-18 10:51:46,686][INFO ][node ] [command
center]
{0.20.1}[8901]: initialized
[2012-12-18 10:51:46,686][INFO ][node ] [command
center]
{0.20.1}[8901]: starting ...
[2012-12-18 10:51:46,879][INFO ][transport ] [command
center]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.6.0.88:9300]}
[2012-12-18 10:51:49,933][INFO ][cluster.service ] [command
center]
new_master [command
center][erqnAKNCQUu_tUyLUSgS6w][inet[/10.6.0.88:9300]]{master=true},
reason:
zen-disco-join (elected_as_master)
[2012-12-18 10:51:49,992][INFO ][discovery ] [command
center]
[server]/erqnAKNCQUu_tUyLUSgS6w
[2012-12-18 10:51:50,042][INFO ][http ] [command
center]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.6.0.88:9200]}
[2012-12-18 10:51:50,043][INFO ][node ] [command
center]
{0.20.1}[8901]: started
[2012-12-18 10:51:50,230][INFO ][gateway ] [command
center]
recovered [0] indices into cluster_state
[2012-12-18 10:54:31,674][WARN ][transport.netty ] [command
center]
exception caught on transport layer [[id: 0x4d0e98fc, /10.6.0.88:58532=>
/10.6.0.88:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message
format
at

org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)

    at

org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)

    at

org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)

    at

org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:786)

    at

org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)

    at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)

    at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)

    at

org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)

    at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)

    at

org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)

    at

org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)

    at

org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)

    at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)

enthought-es.log

--

--
Met vriendelijke groet,

Martijn van Groningen

--

5

Hi Ognen,

I would assume the ES client bound to your logstash is using a different
version. For example the latest (1.1.5) uses ES 0.19.8:

So you can either use ES 0.19.8 for logstash 1.1.15, or you can try
elasticsearch_http:

Martijn asked about your other nodes because logstash's elasticsearch
output uses ES Node Client:

Which is basically another node :slight_smile: And your nodes should run the same
version of ES.

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Wed, Dec 19, 2012 at 6:23 AM, Ognen Duzlevski ognen@enthought.comwrote:

Sorry, I meant to say, I am not running any other nodes, this is just an
experimental setup to see if I can ship syslogs from various machines to
logsash and use a standalone elasticsearch.
Ognen

On Tuesday, December 18, 2012 10:21:01 PM UTC-6, Ognen Duzlevski wrote:

On Tuesday, December 18, 2012 2:05:24 PM UTC-6, Martijn v Groningen wrote:

Are all other nodes also running version 0.20.1?

No other nodes.
OD

On 18 December 2012 18:02, Ognen Duzlevski og...@enthought.com wrote:

I run elasticsearch on a debian server. It starts up, the log is fine
until
I run logstash. Then I start getting errors in the elasticsearch logs
as
below. Any idea what is going on?

Thanks!
Ognen

[2012-12-18 10:51:42,111][INFO ][node ] [command
center]
{0.20.1}[8901]: initializing ...
[2012-12-18 10:51:42,122][INFO ][plugins ] [command
center]
loaded , sites
[2012-12-18 10:51:46,686][INFO ][node ] [command
center]
{0.20.1}[8901]: initialized
[2012-12-18 10:51:46,686][INFO ][node ] [command
center]
{0.20.1}[8901]: starting ...
[2012-12-18 10:51:46,879][INFO ][transport ] [command
center]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.6.0.88:9300]}
[2012-12-18 10:51:49,933][INFO ][cluster.service ] [command
center]
new_master [command
center][erqnAKNCQUu_**tUyLUSgS6w][inet[/10.6.0.88:**9300]]{master=true},
reason:
zen-disco-join (elected_as_master)
[2012-12-18 10:51:49,992][INFO ][discovery ] [command
center]
[server]/erqnAKNCQUu_**tUyLUSgS6w
[2012-12-18 10:51:50,042][INFO ][http ] [command
center]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.6.0.88:9200]}
[2012-12-18 10:51:50,043][INFO ][node ] [command
center]
{0.20.1}[8901]: started
[2012-12-18 10:51:50,230][INFO ][gateway ] [command
center]
recovered [0] indices into cluster_state
[2012-12-18 10:54:31,674][WARN ][transport.netty ] [command
center]
exception caught on transport layer [[id: 0x4d0e98fc, /10.6.0.88:58532=>
/10.6.0.88:9300]], closing connection
java.io.**StreamCorruptedException: invalid internal transport
message format
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.
decode(SizeHeaderFrameDecoder.**java:27)
at
org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.callDecode(**FrameDecoder.java:422)
at
org.elasticsearch.common.netty.handler.codec.frame.
FrameDecoder.messageReceived(**FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.**java:70)
at
org.elasticsearch.common.**netty.channel.DefaultChannelPipeline.
sendUpstream(**DefaultChannelPipeline.java:**558)
at
org.elasticsearch.common.**netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(
DefaultChannelPipeline.java:**786)
at
org.elasticsearch.common.**netty.OpenChannelsHandler.**handleUpstream(
**OpenChannelsHandler.java:74)
at
org.elasticsearch.common.**netty.channel.DefaultChannelPipeline.
sendUpstream(**DefaultChannelPipeline.java:**558)
at
org.elasticsearch.common.**netty.channel.DefaultChannelPipeline.
sendUpstream(**DefaultChannelPipeline.java:**553)
at
org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:268)
at
org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.**java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.read(NioWorker.java:**84)
at
org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.**processSelectedKeys(**AbstractNioWorker.java:471)
at
org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(**AbstractNioWorker.java:332)
at
org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:**35)
at
org.elasticsearch.common.**netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:**102)
enthought-es.log

--

--
Met vriendelijke groet,

Martijn van Groningen

--

--

6

Can you check if your logstash client setup uses port 9200, or 9300? 9200
is the HTTP REST API port for logstash afaik.

Jörg

--

7

Radu,

On Wednesday, December 19, 2012 2:39:57 AM UTC-6, Radu Gheorghe wrote:

Hi Ognen,

I would assume the ES client bound to your logstash is using a different
version. For example the latest (1.1.5) uses ES 0.19.8:
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

So you can either use ES 0.19.8 for logstash 1.1.15, or you can try
elasticsearch_http:
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

Martijn asked about your other nodes because logstash's elasticsearch
output uses ES Node Client:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Which is basically another node :slight_smile: And your nodes should run the same
version of ES.

Thank you, that was it - I installed the 0.19.8 elasticsearch and ran
logstash 1.1.5 and it all works now :slight_smile:
OD

--