This node does not have the remote_cluster_client role

Hi all,

Been going through the doc, the topics and github. Can't seem to find an answer tho this one.

I have 2 clusters running in 2 different regions and wanted to test CCR. Both of them are running on 7.11.1. I was able to setup the remote clusters on both of them and can even do cross cluster searching on them. However, when I try to configure the cross cluster replication, I get the following error in Kibana:

Also tried it through the API:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"this node does not have the remote_cluster_client role"}],"type":"illegal_argument_exception","reason":"this node does not have the remote_cluster_client role"},"status":400}

I checked the logs and even tried setting the logger to trace. Unfortunately, nothing relevant to this comes up.

If I try to query the nodes API on both clusters, this is what I get:

In the ON region:

root@logs-ing-02 ~: curl -nks "https://127.0.0.1:9200/_nodes/remote_cluster_client:true" | jq '._nodes'
{
  "total": 0,
  "successful": 0,
  "failed": 0
}
#=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
root@logs-ing-02 ~: curl -nks "https://127.0.0.1:9200/_nodes/ingest:true" | jq '._nodes'
{
  "total": 2,
  "successful": 2,
  "failed": 0
}
#=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
root@logs-ing-02 ~: curl -nks "https://127.0.0.1:9200/_nodes/ingest:true" | jq -c '.nodes[] | {"name": .name, "roles":.roles}'
{"name":"logs-ing-02.on","roles":["ingest","ml","remote_cluster_client","transform"]}
{"name":"logs-ing-01.on","roles":["ingest","ml","remote_cluster_client","transform"]}

In the QC region:

 root@logs-02 ~: curl -nks "https://127.0.0.1:9200/_nodes/remote_cluster_client:true" | jq '._nodes'
{
  "total": 0,
  "successful": 0,
  "failed": 0
}
#=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
root@logs-02 ~: curl -nks "https://127.0.0.1:9200/_nodes/ingest:true" | jq  '._nodes '
{
  "total": 2,
  "successful": 2,
  "failed": 0
}
#=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
root@logs-02 ~: curl -nks "https://127.0.0.1:9200/_nodes/ingest:true" | jq -c '.nodes[] | {"name": .name, "roles":.roles}'
{"name":"logs-ing-01","roles":["ingest","ml","remote_cluster_client","transform"]}
{"name":"logs-ing-02","roles":["ingest","ml","remote_cluster_client","transform"]}

Both clusters can talk to each other on the http and transport ports and they are using the same CA for their certs.

Has anybody encountered this behaviour before and/or can anyone point me to how I can debug this?

Ok think I figured it out.

Seems like CCR needs the masters to also have the role for some reason. Added the remote_cluster_client role to my masters and now it works.

I'm sorry that you did not have a good experience here. I've opened a PR so that we can clarify this in the docs.

HI Jason,

No worries. I'm still struggling with getting it setup properly, but I just saw an interesting bit in your PR that might fix my issue :slight_smile:

Thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.