Tribe node setup, /_cat/nodes unresponsive

Hi all,

I have a tribe node setup across two clusters, on server0 ($IP0) and
server1 ($IP1), using unicast discovery.
server0 also runs the elasticsearch tribe node instance.

This setup works fine most of the time, and I can see logs from all
clusters on tribe node Kibana.

The issue is that sometimes the elasticsearch instances go unresponsive.
More specifically, it's the nodes API (including /_cat/nodes) that do not
respond, which results in Kibana throwing the following error message:

Could not reach http://.../_nodes. If you are using a proxy, ensure it is

configured correctly

At first, I thought it was just the tribe node that was doing this, but
just today I noticed this on one of the clusters as well.
Other URLs don't seem to be affected (e.g. /, /_cluster/health, etc).

After some time, the nodes API comes back and produces the following
output, and goes unresponsive again.
I've also tried restarting the elasticsearch processes. While it fixes the
unresponsiveness in the short term, it then goes back to the same cycle.

========
jun@server0 ~ % curl -XGET "$IP0:9201/_cat/nodes?pretty=true&v=true"
host ip heap.percent ram.percent load node.role master name
server0 $IP0 11 5 2.97 c x
tribe
server1 $IP1 16 38 2.74 d x
s1-elasticsearch
server0 $IP0 17 5 2.97 d x
s0-elasticsearch
server1 $IP1 c x
s1-logstash
server0 $IP0 c x
tribe
server0 $IP0 c x
tribe
server0 $IP0 c x
tribe
server0 $IP0 c x
s0-logstash
jun@server0 ~ % curl -XGET "$IP0:9200/_cat/nodes?pretty=true&v=true"
{
"error" : "NullPointerException[null]",
"status" : 500
}
jun@server0 ~ % curl -XGET "$IP1:9200/_cat/nodes?pretty=true&v=true"
{
"error" : "NullPointerException[null]",
"status" : 500
}
jun@server0 ~ % curl -XGET '$IP0:9201/_nodes/process?pretty=true&v=true'
{
"cluster_name" : "elasticsearch",
"nodes" : {
"gPX5VOT5TdGPOF6wq9ErKA" : {
"name" : "tribe",
"transport_address" : "inet[/$IP0:9300]",
"host" : "server0",
"ip" : "$IP0",
"version" : "1.3.4",
"build" : "a70f3cc",
"http_address" : "inet[/$IP0:9201]",
"attributes" : {
"client" : "true",
"data" : "false"
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 187050,
"max_file_descriptors" : 65535,
"mlockall" : false
}
},
"liMvaS8cSu2PafCM653LHA" : {
"name" : "s1-elasticsearch",
"transport_address" : "inet[/$IP1:9301]",
"host" : "server1",
"ip" : "$IP1",
"version" : "1.3.4",
"build" : "a70f3cc",
"http_address" : "inet[/$IP1:9200]",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 32465,
"max_file_descriptors" : 65535,
"mlockall" : false
}
},
"z-GOP2v7Q7-FvlaS8bIBJQ" : {
"name" : "s0-elasticsearch",
"transport_address" : "inet[/$IP0:9301]",
"host" : "server0",
"ip" : "$IP0",
"version" : "1.3.4",
"build" : "a70f3cc",
"http_address" : "inet[/$IP0:9200]",
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 165512,
"max_file_descriptors" : 65535,
"mlockall" : false
}
}
}
}

Any help would be greatly appreciated.

Thanks,
Jun

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/314fd788-a883-45c6-9c94-226dd29f9d5c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.