Marvel issue with Elasticsearch 1.4.2 version setup using Oracle Java 1.8.0.25

ajay_bh111 · January 8, 2015, 7:08pm

I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on marvel
nodes without any error . When ES nodes try to connect to monitoring nodes
following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25 on
all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)

[2015-01-08 18:51:05,993][WARN ][transport.netty ] [
es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 => /10.236.54.81:9300]], closing
connectionjava.io.StreamCorruptedException: invalid internal transport
message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)

[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]

Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.

Thanks
Ajay

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3cf77cc5-c206-466e-85d0-d1247a18d733%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ajay_bh111 · January 8, 2015, 8:52pm

Update : If Marvel deployed on data cluster itself i.e. data not shipped
to monitoring nodes, it works fine.
Thanks
Ajay

On Thursday, January 8, 2015 at 2:08:10 PM UTC-5, ajay....@gmail.com wrote:

I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on
marvel nodes without any error . When ES nodes try to connect to monitoring
nodes following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25
on all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)

[2015-01-08 18:51:05,993][WARN ][transport.netty ] [
es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 http://10.236.54.121:52237 => /10.236.54.81:9300]],
closing connectionjava.io.StreamCorruptedException: invalid internal
transport message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)

[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]

Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.

Thanks
Ajay

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d0778bae-f11f-4cb0-a287-77b67c350349%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Boaz_Leskes · January 11, 2015, 8:53am

Did you configure marvel to send data to port 9300? Marvel uses http and the rest API for its data shipping and this need to be configured to use port 9200 (assuming defaults)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/14ce8a00-acb0-4cba-8483-1367dad6c1f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ajay_bh111 · January 11, 2015, 2:21pm

Ye‎s, it was upgraded from working 1.3.2 version setup earlier but encountered same errors then tried to go to all fresh setup but with same results.

Thanks
Ajay

Sent from my BlackBerry 10 smartphone on the Rogers network.
Original Message
From: Boaz Leskes‎
Sent: Sunday, January 11, 2015 03:53
To: elasticsearch@googlegroups.com
Reply To: elasticsearch@googlegroups.com
Subject: Re: Marvel issue with Elasticsearch 1.4.2 version setup using Oracle Java 1.8.0.25

Did you configure marvel to send data to port 9300? Marvel uses http and the rest API for its data shipping and this need to be configured to use port 9200 (assuming defaults)

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Q7er2OlexHE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/14ce8a00-acb0-4cba-8483-1367dad6c1f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150111142120.31154262.5230.1118%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

ajay_bh111 · January 11, 2015, 6:55pm

Here are the settings used :

On data nodes :
marvel.agent.exporter.es.hosts: ["es-orn-mon-01:9300","es-orn-mon-02:9300"]

On Marvel nodes :
discovery.zen.ping.unicast.hosts: [
"es-orn-mon-01:9300","es-orn-mon-02:9300" ]

Ajay

On Thursday, January 8, 2015 at 2:08:10 PM UTC-5, ajay....@gmail.com wrote:

I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on
marvel nodes without any error . When ES nodes try to connect to monitoring
nodes following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25
on all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)

[2015-01-08 18:51:05,993][WARN ][transport.netty ] [
es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 http://10.236.54.121:52237 => /10.236.54.81:9300]],
closing connectionjava.io.StreamCorruptedException: invalid internal
transport message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)

[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]

Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.

Thanks
Ajay

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47926e98-4e28-45b1-b290-e7d871ade92b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Boaz_Leskes · January 11, 2015, 9:31pm

Heya,

You should switch to port 9200 in your marvel.agent settings. 9300 is fine for the unicast ones.

Cheers,

Boaz

On Sun, Jan 11, 2015 at 10:55 AM, null ajay.bh111@gmail.com wrote:

Here are the settings used :
On data nodes :
marvel.agent.exporter.es.hosts: ["es-orn-mon-01:9300","es-orn-mon-02:9300"]
On Marvel nodes :
discovery.zen.ping.unicast.hosts: [
"es-orn-mon-01:9300","es-orn-mon-02:9300" ]
Ajay
On Thursday, January 8, 2015 at 2:08:10 PM UTC-5, ajay....@gmail.com wrote:

I am trying to set up monitoring cluster of marvel in test setup using new
elasticsearch 1.4.2 and latest marvel (pulled with bin/plugin -i
elasticsearch/marvel/latest command) . without marvel ES starts on
marvel nodes without any error . When ES nodes try to connect to monitoring
nodes following error is thrown on Marvel nodes : (java vesrion is 1.8.0.25
on all nodes). All nodes same OS Ubuntu 3.13.0-40-generic x64)

[2015-01-08 18:51:05,993][WARN ][transport.netty ] [
es-orn-mon-02] exception caught on transport layer [[id: 0xf67ca3df,
/10.236.54.121:52237 http://10.236.54.121:52237 => /10.236.54.81:9300]],
closing connectionjava.io.StreamCorruptedException: invalid internal
transport message format, got (47,45,54,20)
at
org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:47)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On ES cluster nodes following error is thrown : (cluster nodes are able to
connect on port 9300 of monitoring nodes)

[2015-01-08 18:57:10,545][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-01:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
failed to verify/upload the marvel template to [es-orn-mon-02:9300]:
Unexpected end of file from server
[2015-01-08 18:57:10,550][ERROR][marvel.agent.exporter ] [es-orn-c-01]
could not connect to any configured elasticsearch instances:
[es-orn-mon-01:9300,es-orn-mon-02:9300]

Elasticsearch alone works fine on both monitoring and data cluster if
marvel plugin is removed.

Thanks
Ajay

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Q7er2OlexHE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/47926e98-4e28-45b1-b290-e7d871ade92b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1421011868040.313e0ae6%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.