Setting up a Separate Monitoring Cluster


(Geeshan S) #1

Hi All,

I'm trying to setup a separate cluster(kibanacluster) for monitoring my primary elasticsearch cluster (marveltest) . Below are the ES, Marvel and Kibana versions I'm using. The ES version is fixed for the moment. I can update or downgrade the other components if needed.


The monitoring cluster and Kibana are both running in the host and the primary cluster is running in a separate host elasticsearch.yml has the below entry [""]

marvel.enabled: true

type: http
host: [""]

Looking at the DEBUG logs in the monitoring cluster i can see data is coming from the primary cluster but is getting "filtered" since the cluster name is different.

[2016-07-04 16:33:25,144][DEBUG][transport.netty ] [nodek] connected to node [{#zen_unicast_2#}{}{}]
[2016-07-04 16:33:25,144][DEBUG][transport.netty ] [nodek] connected to node [{#zen_unicast_1#}{}{}]
[2016-07-04 16:33:25,183][DEBUG][] [nodek] [1] filtering out response from {node1}{Rmgg0Mw1TSmIpytqfnFgFQ}{}{}, not same cluster_name [marveltest]
[2016-07-04 16:33:26,533][DEBUG][] [nodek] [1] filtering out response from {node1}{Rmgg0Mw1TSmIpytqfnFgFQ}{}{}, not same cluster_name [marveltest]
[2016-07-04 16:33:28,039][DEBUG][] [nodek] [1] filtering out response from {node1}{Rmgg0Mw1TSmIpytqfnFgFQ}{}{}, not same cluster_name [marveltest]
[2016-07-04 16:33:28,040][DEBUG][transport.netty ] [nodek] disconnecting from [{#zen_unicast_2#}{}{}] due to explicit disconnect call
[2016-07-04 16:33:28,040][DEBUG][discovery.zen ] [nodek] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [{nodek}{vQ-Iq8dKSz26AJUX77Ncfw}{}{}], id[42], master [{nodek}{vQ-Iq8dKSz26AJUX77Ncfw}{}{}], hasJoinedOnce [true], cluster_name[kibanacluster]}
[2016-07-04 16:33:28,053][DEBUG][transport.netty ] [nodek] disconnecting from [{#zen_unicast_1#}{}{}] due to explicit disconnect call
[2016-07-04 16:33:28,057][DEBUG][transport.netty ] [nodek] connected to node [{nodek}{vQ-Iq8dKSz26AJUX77Ncfw}{}{}]
[2016-07-04 16:33:28,117][DEBUG][discovery.zen.publish ] [nodek] received full cluster state version 32 with size 5589

Thank you,

(Mark Walkom) #2

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

Did you follow the docs, here -

(Geeshan S) #3

Sorry about that. I have edited the post.
Yes I followed the above link in setting it up.
Could this be a licence issue? Since there are two clusters involved?

(Mark Walkom) #4

It's not a license issue.

Can you show the elasticsearch.yml file of each (but remove comments, and empty lines).

(Geeshan S) #5

Hi Mark,

Below are the elasticsearch.yml files.

2.116 (Primary cluster) marveltest node1
bootstrap.mlockall: true ["", ""]
gateway.recover_after_nodes: 1
gateway.expected_nodes: 1
gateway.recover_after_time: 1m
path.repo: ["/usr/local/surf/ES_BACKUP"]
script.inline: true
script.indexed: true

2.124(Monitoring Cluster) kibanacluster nodek
node.master: false false ["", ""]

(Mark Walkom) #6

I'd suggest you read, as I mentioned.

However you don't point each cluster discovery at the other node.

(system) #7