I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on marvel
nodes without any error . When ES nodes try to connect to monitoring nodes
following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25 on
all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)
[2015-01-08 18:51:05,993][WARN ][transport.netty ] [ es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 => /10.236.54.81:9300]], closing
connectionjava.io.StreamCorruptedException: invalid internal transport
message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)
[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]
Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.
Update : If Marvel deployed on data cluster itself i.e. data not shipped
to monitoring nodes, it works fine.
Thanks
Ajay
On Thursday, January 8, 2015 at 2:08:10 PM UTC-5, ajay....@gmail.com wrote:
I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on
marvel nodes without any error . When ES nodes try to connect to monitoring
nodes following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25
on all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)
[2015-01-08 18:51:05,993][WARN ][transport.netty ] [ es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 http://10.236.54.121:52237 => /10.236.54.81:9300]],
closing connectionjava.io.StreamCorruptedException: invalid internal
transport message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)
[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]
Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.
Did you configure marvel to send data to port 9300? Marvel uses http and the rest API for its data shipping and this need to be configured to use port 9200 (assuming defaults)
Yes, it was upgraded from working 1.3.2 version setup earlier but encountered same errors then tried to go to all fresh setup but with same results.
Thanks
Ajay
Sent from my BlackBerry 10 smartphone on the Rogers network.
Original Message
From: Boaz Leskes
Sent: Sunday, January 11, 2015 03:53
To: elasticsearch@googlegroups.com
Reply To: elasticsearch@googlegroups.com
Subject: Re: Marvel issue with Elasticsearch 1.4.2 version setup using Oracle Java 1.8.0.25
Did you configure marvel to send data to port 9300? Marvel uses http and the rest API for its data shipping and this need to be configured to use port 9200 (assuming defaults)
On data nodes :
marvel.agent.exporter.es.hosts: ["es-orn-mon-01:9300","es-orn-mon-02:9300"]
On Marvel nodes :
discovery.zen.ping.unicast.hosts: [
"es-orn-mon-01:9300","es-orn-mon-02:9300" ]
Ajay
On Thursday, January 8, 2015 at 2:08:10 PM UTC-5, ajay....@gmail.com wrote:
I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on
marvel nodes without any error . When ES nodes try to connect to monitoring
nodes following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25
on all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)
[2015-01-08 18:51:05,993][WARN ][transport.netty ] [ es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 http://10.236.54.121:52237 => /10.236.54.81:9300]],
closing connectionjava.io.StreamCorruptedException: invalid internal
transport message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)
[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]
Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.
Here are the settings used :
On data nodes :
marvel.agent.exporter.es.hosts: ["es-orn-mon-01:9300","es-orn-mon-02:9300"]
On Marvel nodes :
discovery.zen.ping.unicast.hosts: [
"es-orn-mon-01:9300","es-orn-mon-02:9300" ]
Ajay
On Thursday, January 8, 2015 at 2:08:10 PM UTC-5, ajay....@gmail.com wrote:
I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on
marvel nodes without any error . When ES nodes try to connect to monitoring
nodes following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25
on all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)
[2015-01-08 18:51:05,993][WARN ][transport.netty ] [ es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 http://10.236.54.121:52237 => /10.236.54.81:9300]],
closing connectionjava.io.StreamCorruptedException: invalid internal
transport message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)
[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]
Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.