Tribe Node

Elastic Stack Elasticsearch
1

I just have a few questions about the mechanics of a tribe node.

I have set up two clusters, and a tribe node configured to connect to the two of them. First, how can I verify that it is successfully connected. I can run curl commands, but the health check and cluster state checks return only data for the single node (as I believe they were supposed to according to the tribe node documentation). What is a good check to verify connection.

Secondly, I am trying to set up a kibana instance to talk to this node, so that it can search across clusters, and I am getting the errors:

log   [20:27:47.739] [info][status][plugin:elasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log   [20:28:17.757] [error][status][plugin:elasticsearch] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log   [20:28:25.286] [info][status][plugin:elasticsearch] Status changed from red to yellow - No existing Kibana index found
log   [20:28:55.293] [error][status][plugin:elasticsearch] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log   [20:29:02.833] [info][status][plugin:elasticsearch] Status changed from red to yellow - No existing Kibana index found
log   [20:29:32.837] [error][status][plugin:elasticsearch] Status changed from yellow to red - Unable to create Kibana index ".kibana"
log   [20:29:40.362] [info][status][plugin:elasticsearch] Status changed from red to yellow - No existing Kibana index found
log   [20:30:10.369] [error][status][plugin:elasticsearch] Status changed from yellow to red - Unable to create Kibana index ".kibana"

I assume this is because this is a new kibana node, and it typically creates an index for itself on the cluster to store data in. What is the logical home location for this index if it is connecting to a tribe node?? Should the tribe node be node.data: true?? What if I would like to have a kibana node that queries the tribe node for cross-cluster queries, but also would like to have individual kibana nodes connected to each cluster. Is there a way to configure this?? I can see how it could be a problem because the .kibana index would then exist on multiple clusters, which it appears tribe nodes do not handle well.

Lastly, the documentation states that The tribe node will create a node client to connect each cluster using unicast discovery by default. Does this mean, that if I run a cluster health check on one of the clusters before and after i turn on the tribe node that it should actually show another node added to the cluster??

Thank you!!!

2
  1. Use the _cat REST api with your tribe node.

E.g.
$ curl tribenode:9200/_cat/nodes?v

That will list all the nodes that your tribe knows about.

If you use that on one of your downstream clusters/nodes, it should show your tribe connected as a client.

2 - The problem is tribe (as a client) can not create new index. It can only read an existing one. So you will have to pre-create your kibana index first. See the following blog post.

Using kibana with tribe nodes

3

Ah thank you! That helped ton! Do you know if it is an option to set the tribe node as node.data: true, and then let it host it's own .kibana index?? I will configure the other nodes to use a different index name so they do not conflict, but I would like to do that simply because there is no logical better cluster to put the tribe node's kibana index on.

4

Nope, you can't use a tribe as a data node. Tribe is a client node, not master and not data.

There are several solutions. Since I have many downstream clusters (100+ clusters and over 1200 nodes), I want dedicated cluster for tribe kibana index. For smaller setup, you could potentially run 2 ES on one host. One for tribe and one for the kibana index only.

For redundancy, run 2 or more tribes and 2 or more dedicated kibana index ES. Put a proxy in front. I use nginx and ATS.

You could also put the tribe kibana index in one of your downstream cluster and use a tool to copy/sync it to all other downstream clusters. Use the

on_conflict: prefer_cluster

settings in your elasticsearch.yml to point to one that you pick as the "master" copy of your kibana index. Sync that to all others.

I have automated scripts that do this and backup kibana indices for me.

Tin

1 Like
5

Ahh yeah, I see that's what they did in the tutorial too, a dedicated configuration cluster, that's great. Thanks!