Marvel monitoring on seperate cluster - connect timed out

Hi,

I have a problem running Marvel on a dedicated monitoring cluster. Es
1.0.0RC2. Whan i start a node with marvel.agent.exporter.es.hosts set to
localhost:9200 - work fine ... but when
setting marvel.agent.exporter.es.hosts to ["es-mon-1.z:9200"] i get

[2014-02-05 14:05:16,459][ERROR][marvel.agent.exporter ] could not
connect to any configured elasticsearch instances: [es-mon-1.z:9200]
[2014-02-05 14:05:17,273][ERROR][marvel.agent.exporter ] error
connecting to [es-mon-1.z:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:995)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:931)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:849)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.openConnection(ESExporter.java:317)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.openConnection(ESExporter.java:293)
at
org.elasticsearch.marvel.agent.exporter.ESExporter.access$200(ESExporter.java:56)
at
org.elasticsearch.marvel.agent.exporter.ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

There is a working connection from the data node to the monitoring node.

PING es-mon-1.z: 56 data bytes
64 bytes from 192.168.0.202: icmp_seq=0. time=0.097 ms
64 bytes from 192.168.0.202: icmp_seq=1. time=0.095 ms
64 bytes from 192.168.0.202: icmp_seq=2. time=0.094 ms
^C
----es-mon-1.z PING Statistics----
3 packets transmitted, 3 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.094/0.095/0.097/0.002

[root@es-d-2 /opt/es/bin]# curl es-mon-1.z:9200
{
"status" : 200,
"name" : "es-mon-1",
"version" : {
"number" : "1.0.0.RC2",
"build_hash" : "a9d736e0e77ffd046a10500c779d574c6960f704",
"build_timestamp" : "2014-02-03T15:02:11Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
},
"tagline" : "You Know, for Search"
}

There are no error logs on the monitoring cluster running Marvel plugin.

[2014-02-05 14:08:16,987][INFO ][node ] [es-mon-1]
started
[2014-02-05 14:08:18,088][INFO ][monitor.jvm ] [es-mon-1]
[gc][young][5][1] duration [911ms], collections [1]/[1s], total
[911ms]/[911ms], memory [251.3mb]->[39.9mb]/[989.8mb], all_pools {[young]
[251.3mb]->[5.7mb]/[273mb]}{[survivor] [0b]->[34.1mb]/[34.1mb]}{[old]
[0b]->[132.6kb]/[682.6mb]}
[2014-02-05 14:08:24,350][INFO ][node ] [es-mon-1]
stopping ...
[2014-02-05 14:08:24,370][INFO ][node ] [es-mon-1]
stopped
[2014-02-05 14:08:24,371][INFO ][node ] [es-mon-1]
closing ...
[2014-02-05 14:08:24,376][INFO ][node ] [es-mon-1]
closed
[2014-02-05 14:08:25,629][INFO ][node ] [es-mon-1]
version[1.0.0.RC2], pid[26859], build[a9d736e/2014-02-03T15:02:11Z]
[2014-02-05 14:08:25,630][INFO ][node ] [es-mon-1]
initializing ...
[2014-02-05 14:08:25,642][INFO ][plugins ] [es-mon-1]
loaded [marvel], sites [marvel, head, browser, segmentspy, paramedic]
[2014-02-05 14:08:28,335][INFO ][marvel.agent ] [es-mon-1]
collecting disabled by settings
[2014-02-05 14:08:28,509][INFO ][node ] [es-mon-1]
initialized
[2014-02-05 14:08:28,510][INFO ][node ] [es-mon-1]
starting ...
[2014-02-05 14:08:28,592][INFO ][transport ] [es-mon-1]
bound_address {inet[/192.168.0.202:9300]}, publish_address
{inet[/192.168.0.202:9300]}
[2014-02-05 14:08:31,739][INFO ][cluster.service ] [es-mon-1]
new_master
[es-mon-1][LXesyAHIRoaBybJnkWcylQ][es-mon-1.z][inet[/192.168.0.202:9300]]{machine=44454c4c-4d00-1046-804a-c4c04f365831,
master=true}, reason: zen-disco-join (elected_as_master)
[2014-02-05 14:08:31,748][INFO ][discovery ] [es-mon-1]
monitoring/LXesyAHIRoaBybJnkWcylQ
[2014-02-05 14:08:31,796][INFO ][http ] [es-mon-1]
bound_address {inet[/192.168.0.202:9200]}, publish_address
{inet[/192.168.0.202:9200]}
[2014-02-05 14:08:32,525][INFO ][gateway ] [es-mon-1]
recovered [1] indices into cluster_state
[2014-02-05 14:08:32,526][INFO ][node ] [es-mon-1]
started
[2014-02-05 14:08:34,059][WARN ][monitor.jvm ] [es-mon-1]
[gc][young][5][1] duration [1.3s], collections [1]/[1.5s], total
[1.3s]/[1.3s], memory [256.8mb]->[42.9mb]/[989.8mb], all_pools {[young]
[256.8mb]->[8.5mb]/[273mb]}{[survivor] [0b]->[34.1mb]/[34.1mb]}{[old]
[0b]->[302kb]/[682.6mb]}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dacc218d-c214-4638-bb6a-1650d408bf7e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/25351002-4e66-4f8b-a2f5-dab0ba306103%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Paweł,

Trying to pin point what it can be - can you use the ip 192.168.0.202
instead of host name in the marvel.agent.exporter.es.hosts setting?

Cheers,
Boaz

On Wednesday, February 5, 2014 2:10:46 PM UTC+1, Paweł Krzaczkowski wrote:

java version "1.7.0_25"

Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/70a624c5-48a7-4347-839b-5ab9ff2787da%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.
java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(
HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.openConnection
(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.openConnection
(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.access$200(
ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/25bb469f-2eb3-42ed-b75d-e8c7fbd52c4a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
http://curl.haxx.se/docs/manpage.html

If so, you can tell ES to use the same by adding the following properties
upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(
HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.access$200(
ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4a1df5c2-4161-4566-a409-acf01eb35f6f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

There is no proxy .. it's a direct connection ... two machines connected to
the same switch ... same network / same network settings.

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following properties
upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(
HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.access$200(
ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f2ec1f6e-050b-4972-8e58-9cd0020ee526%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Even better .. if I set cluster name on none 192.168.0.202 to my default
cluster name - the node will join the cluster and the connection is working
fine. The only thing that is not working is if i change the cluster name to
"monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following properties
upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(
HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.access$200(
ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/067eba97-8d27-4fda-b923-5b5415d5b612%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my default
cluster name - the node will join the cluster and the connection is working
fine. The only thing that is not working is if i change the cluster name to
"monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following properties
upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(
HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.access$200
(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e3e89fe8-37f0-4301-b830-ce3c2d541256%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[root@es-d-2 ~]# env
MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my default
cluster name - the node will join the cluster and the connection is working
fine. The only thing that is not working is if i change the cluster name to
"monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski
wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient
(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5b1cc553-6663-4e49-9ff3-5b95edfcb690%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'

Note that ES will fail this request as it is not valid but I want to double
check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my default
cluster name - the node will join the cluster and the connection is working
fine. The only thing that is not working is if i change the cluster name to
"monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski
wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/38c6336b-de58-4f7d-b3ef-5c23e8da97b5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski
wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski
wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could notconnect to any configured elasticsearch instances
: [192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378
)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473
)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0c4094ed-1770-4757-b287-b9d0f9a43ad6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł Krzaczkowski
napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski
wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski
wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could
not connect to any configured elasticsearch instances: [192.168.
0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392
)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:
378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:
473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/30329fbb-a827-4828-9983-3cef60806ed1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ok it looks like it has something to do with the system it's running on.

Both es-d-1 and es-mon-1 run on smartos Solaris - and it doesn't work. But
I just started another data node on linux -> and this node connects to
es-mon-1.z - well it gives one 'connect timed out' error - but then it
establishes a working connection.

W dniu czwartek, 6 lutego 2014 13:28:29 UTC+1 użytkownik Paweł Krzaczkowski
napisał:

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test
body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski
wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski
wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could
not connect to any configured elasticsearch instances: [192.168.
0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:
392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:
378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:
473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect
(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/adcc563e-732e-4268-984e-b9cf96e68ec6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[root@es-mon-1 /opt/es]# uname -a
SunOS es-mon-1 5.11 joyent_20130508T120818Z i86pc i386 i86pc Solaris

W dniu czwartek, 6 lutego 2014 13:40:00 UTC+1 użytkownik Paweł Krzaczkowski
napisał:

Ok it looks like it has something to do with the system it's running on.

Both es-d-1 and es-mon-1 run on smartos Solaris - and it doesn't work. But
I just started another data node on linux -> and this node connects to
es-mon-1.z - well it gives one 'connect timed out' error - but then it
establishes a working connection.

W dniu czwartek, 6 lutego 2014 13:28:29 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test
body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski
wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski
wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł Krzaczkowski
wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ] could
not connect to any configured elasticsearch instances: [192.168.
0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ] error
connecting to [192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:
392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:
378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:
473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.
plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/643ce10b-cb44-40b8-9043-5711b79caa1a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

OK i don;t know how .. but after 10 min of error it connected but it looks
like it's not sending all the information - maybe Marvel is not able to
collect system information from Solaris (smartOS)

https://picasaweb.google.com/lh/photo/L9Jptbfzxmh4JL6yVditfsi8TTKXUxXi6TYOgY7UZe8

W dniu czwartek, 6 lutego 2014 13:40:42 UTC+1 użytkownik Paweł Krzaczkowski
napisał:

[root@es-mon-1 /opt/es]# uname -a
SunOS es-mon-1 5.11 joyent_20130508T120818Z i86pc i386 i86pc Solaris

W dniu czwartek, 6 lutego 2014 13:40:00 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Ok it looks like it has something to do with the system it's running on.

Both es-d-1 and es-mon-1 run on smartos Solaris - and it doesn't work.
But I just started another data node on linux -> and this node connects to
es-mon-1.z - well it gives one 'connect timed out' error - but then it
establishes a working connection.

W dniu czwartek, 6 lutego 2014 13:28:29 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test
body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski
wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski
wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł
Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ]could
not connect to any configured elasticsearch instances: [192.168.
0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ]error connecting to
[192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:
392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175
)
at sun.net.www.http.HttpClient.openServer(HttpClient.java
:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java
:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203
)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.
plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e62a3b7a-cf51-46ca-991d-9a7bbf88f9f6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Weird. The missing info is standard ES and should be there independent of
the OS (and solaris is supported, but not relevant here :)).

Double checking - do you access the marvel ui through es-mon-1?

On Thursday, February 6, 2014 1:56:06 PM UTC+1, Paweł Krzaczkowski wrote:

OK i don;t know how .. but after 10 min of error it connected but it looks
like it's not sending all the information - maybe Marvel is not able to
collect system information from Solaris (smartOS)

https://picasaweb.google.com/lh/photo/L9Jptbfzxmh4JL6yVditfsi8TTKXUxXi6TYOgY7UZe8

W dniu czwartek, 6 lutego 2014 13:40:42 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 /opt/es]# uname -a
SunOS es-mon-1 5.11 joyent_20130508T120818Z i86pc i386 i86pc Solaris

W dniu czwartek, 6 lutego 2014 13:40:00 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Ok it looks like it has something to do with the system it's running on.

Both es-d-1 and es-mon-1 run on smartos Solaris - and it doesn't work.
But I just started another data node on linux -> and this node connects to
es-mon-1.z - well it gives one 'connect timed out' error - but then it
establishes a working connection.

W dniu czwartek, 6 lutego 2014 13:28:29 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test
body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski
wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz Leskes
napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł Krzaczkowski
wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default: see
curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł
Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ]could
not connect to any configured elasticsearch instances: [192.168.
0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ]error connecting to
[192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java
:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:
175)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:
203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.
plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/43c91e89-cabb-4874-bf0f-8cb5212f8c5c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yes

http://192.168.0.202:9200/_plugin/marvel

W dniu czwartek, 6 lutego 2014 14:33:07 UTC+1 użytkownik Boaz Leskes
napisał:

Weird. The missing info is standard ES and should be there independent of
the OS (and solaris is supported, but not relevant here :)).

Double checking - do you access the marvel ui through es-mon-1?

On Thursday, February 6, 2014 1:56:06 PM UTC+1, Paweł Krzaczkowski wrote:

OK i don;t know how .. but after 10 min of error it connected but it
looks like it's not sending all the information - maybe Marvel is not able
to collect system information from Solaris (smartOS)

https://picasaweb.google.com/lh/photo/L9Jptbfzxmh4JL6yVditfsi8TTKXUxXi6TYOgY7UZe8

W dniu czwartek, 6 lutego 2014 13:40:42 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 /opt/es]# uname -a
SunOS es-mon-1 5.11 joyent_20130508T120818Z i86pc i386 i86pc Solaris

W dniu czwartek, 6 lutego 2014 13:40:00 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Ok it looks like it has something to do with the system it's running on.

Both es-d-1 and es-mon-1 run on smartos Solaris - and it doesn't work.
But I just started another data node on linux -> and this node connects to
es-mon-1.z - well it gives one 'connect timed out' error - but then it
establishes a working connection.

W dniu czwartek, 6 lutego 2014 13:28:29 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test
body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski
wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz
Leskes napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł
Krzaczkowski wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default:
see curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł
Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ]could
not connect to any configured elasticsearch instances: [192.168
.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ]error connecting to
[192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method
)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.
java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:
175)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:
203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.
plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/116bd3b6-60d6-4339-a8d4-c6530864fea6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

It looks like it connected only once because it still gives errors every 5
minutes

[2014-02-06 14:57:27,268][ERROR][marvel.agent.exporter ] error
connecting to [es-mon-1.z:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.
java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(
HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.openConnection
(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.openConnection
(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.access$200(
ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

W dniu czwartek, 6 lutego 2014 14:42:43 UTC+1 użytkownik Paweł Krzaczkowski
napisał:

Yes

http://192.168.0.202:9200/_plugin/marvel

W dniu czwartek, 6 lutego 2014 14:33:07 UTC+1 użytkownik Boaz Leskes
napisał:

Weird. The missing info is standard ES and should be there independent of
the OS (and solaris is supported, but not relevant here :)).

Double checking - do you access the marvel ui through es-mon-1?

On Thursday, February 6, 2014 1:56:06 PM UTC+1, Paweł Krzaczkowski wrote:

OK i don;t know how .. but after 10 min of error it connected but it
looks like it's not sending all the information - maybe Marvel is not able
to collect system information from Solaris (smartOS)

https://picasaweb.google.com/lh/photo/L9Jptbfzxmh4JL6yVditfsi8TTKXUxXi6TYOgY7UZe8

W dniu czwartek, 6 lutego 2014 13:40:42 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 /opt/es]# uname -a
SunOS es-mon-1 5.11 joyent_20130508T120818Z i86pc i386 i86pc Solaris

W dniu czwartek, 6 lutego 2014 13:40:00 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Ok it looks like it has something to do with the system it's running
on.

Both es-d-1 and es-mon-1 run on smartos Solaris - and it doesn't work.
But I just started another data node on linux -> and this node connects to
es-mon-1.z - well it gives one 'connect timed out' error - but then it
establishes a working connection.

W dniu czwartek, 6 lutego 2014 13:28:29 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk
-d'test body'

Note that ES will fail this request as it is not valid but I want
to double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski
wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz
Leskes napisał:

Hi Pawel,

The only thing I can think of is that potentially some java
system properties or ENV variables get in the way - things like forcing it
use a proxy which I mentioned earlier. Can you list the ENV variables and
the exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł
Krzaczkowski wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz
Leskes napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default:
see curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł
Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ]could
not connect to any configured elasticsearch instances: [
192.168.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ]error connecting to
[192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native
Method)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.
java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:
175)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:
203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290
)
at sun.net.www.http.HttpClient.New(HttpClient.java:306
)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.
plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect
(HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c3ac24bf-50b4-4707-b3d9-3636d274bc6f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

Marvel 1.1.1 fixes the problem ..

W dniu czwartek, 6 lutego 2014 14:33:07 UTC+1 użytkownik Boaz Leskes
napisał:

Weird. The missing info is standard ES and should be there independent of
the OS (and solaris is supported, but not relevant here :)).

Double checking - do you access the marvel ui through es-mon-1?

On Thursday, February 6, 2014 1:56:06 PM UTC+1, Paweł Krzaczkowski wrote:

OK i don;t know how .. but after 10 min of error it connected but it
looks like it's not sending all the information - maybe Marvel is not able
to collect system information from Solaris (smartOS)

https://picasaweb.google.com/lh/photo/L9Jptbfzxmh4JL6yVditfsi8TTKXUxXi6TYOgY7UZe8

W dniu czwartek, 6 lutego 2014 13:40:42 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 /opt/es]# uname -a
SunOS es-mon-1 5.11 joyent_20130508T120818Z i86pc i386 i86pc Solaris

W dniu czwartek, 6 lutego 2014 13:40:00 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Ok it looks like it has something to do with the system it's running on.

Both es-d-1 and es-mon-1 run on smartos Solaris - and it doesn't work.
But I just started another data node on linux -> and this node connects to
es-mon-1.z - well it gives one 'connect timed out' error - but then it
establishes a working connection.

W dniu czwartek, 6 lutego 2014 13:28:29 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

Sorry wrong node :wink:

[root@es-d-2 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}[root@es-d-2
~]#
[root@es-d-2 ~]#

W dniu czwartek, 6 lutego 2014 13:24:16 UTC+1 użytkownik Paweł
Krzaczkowski napisał:

[root@es-mon-1 ~]# curl -XPOST es-mon-1.z:9200/_bulk -d'test body'
{"error":"Failed to derive xcontent from
org.elasticsearch.common.bytes.ChannelBufferBytesReference@b628f070"}
[root@es-mon-1 ~]#

W dniu czwartek, 6 lutego 2014 13:23:03 UTC+1 użytkownik Boaz Leskes
napisał:

Looks good. Can you run the following? (running out of ideas here):

[root@es-d-2 /opt/es/bin]# curl -XPOST es-mon-1.z:9200/_bulk -d'test
body'

Note that ES will fail this request as it is not valid but I want to
double check we get an ES specific error (i.e., it connected).

On Thursday, February 6, 2014 1:05:42 PM UTC+1, Paweł Krzaczkowski
wrote:

[root@es-d-2 ~]# env

MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
HZ=100
SHELL=/usr/bin/bash
TERM=xterm
ES_HEAP_SIZE=8192M
ES_JAVA_OPTS=-d64 -server -Des.processors=4 -Des.node.name=es-d-2
-Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
COLUMNS=238
PAGER=less
MAIL=/var/mail/root

PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
PWD=/root
JAVA_HOME=/opt/jdk1.7.0_25/
LINES=74
SHLVL=1
HOME=/root
TERMINFO=/opt/local/share/lib/terminfo
LOGNAME=root
FTPMODE=auto

Start command:
/opt/es/bin/elasticsearch -d -p /export/es/logs/elasticsearch.pid

W dniu czwartek, 6 lutego 2014 13:02:37 UTC+1 użytkownik Boaz
Leskes napisał:

Hi Pawel,

The only thing I can think of is that potentially some java system
properties or ENV variables get in the way - things like forcing it use a
proxy which I mentioned earlier. Can you list the ENV variables and the
exact command line that is used to start Elasticsearch?

Cheers,
Boaz

On Wednesday, February 5, 2014 3:36:23 PM UTC+1, Paweł
Krzaczkowski wrote:

Even better .. if I set cluster name on none 192.168.0.202 to my
default cluster name - the node will join the cluster and the connection is
working fine. The only thing that is not working is if i change the cluster
name to "monitoring" and try to send Marvel data to this cluster node.

Paweł

W dniu środa, 5 lutego 2014 14:51:59 UTC+1 użytkownik Boaz Leskes
napisał:

Perhaps you have a proxy configured?

curl listens to a couple of environment variables by default:
see curl - How To Use

If so, you can tell ES to use the same by adding the following
properties upon startup: -Dhttp.proxyHost=webcache.example.com

-Dhttp.proxyPort=8080

On Wednesday, February 5, 2014 2:26:06 PM UTC+1, Paweł
Krzaczkowski wrote:

Same error

[2014-02-05 14:25:17,254][ERROR][marvel.agent.exporter ]could
not connect to any configured elasticsearch instances: [192.168
.0.202:9200]
[2014-02-05 14:25:18,204][ERROR][marvel.agent.exporter ]error connecting to
[192.168.0.202:9200]
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method
)
at java.net.AbstractPlainSocketImpl.doConnect(
AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(
AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.
java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:
175)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.
java:473)
at sun.net.www.http.HttpClient.(HttpClient.java:
203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.
getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.
plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(
HttpURLConnection.java:849)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:317)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
openConnection(ESExporter.java:293)
at org.elasticsearch.marvel.agent.exporter.ESExporter.
access$200(ESExporter.java:56)
at org.elasticsearch.marvel.agent.exporter.
ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:734)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b587d4fa-cccd-4941-8ac0-b5352e9b4001%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.