Alerting throws "node [es-client1] does not have the [remote_cluster_client] role"

Hi all,

My cluster consists of the following nodes:

2x coordinating node (empty node.roles)
3x master (node.roles = ['master'])
5x data (node.roles = ['data', 'ingest', 'ml', 'transform'])

I have recently upgraded from 6.8.14 to 7.11.2, which went fairly smooth. The only issue is that Kibana alerting keeps throwing the following errors in the log:

{"type":"log","@timestamp":"2021-03-12T09:46:49+00:00","tags":["error","plugins","alerts","plugins","alerting"],"pid":18141,"message":"Executing Alert \"b0177aa0-8316-11eb-a876-59c2aec22f7c\" has resulted in Error: [illegal_argument_exception] node [es-client1] does not have the [remote_cluster_client] role"}
{"type":"log","@timestamp":"2021-03-12T09:46:49+00:00","tags":["error","plugins","alerts","plugins","alerting"],"pid":18141,"message":"Executing Alert \"b011fc60-8316-11eb-a876-59c2aec22f7c\" has resulted in Error: [illegal_argument_exception] node [es-client2] does not have the [remote_cluster_client] role"}

I am not using CCR so i was under the impression that none of my nodes required the remote_cluster_client role. Is this something that is required for Kibana monitoring/alerting to function?

Google was not really helpful, so i hope someone here can advise. Thanks!

1 Like

This could be related to cross cluster search, and not cross cluster replication.

Is there an alert defined that attempts to search to a remote cluster?

As far as i can see there isn't, its a fairly stock installation after an upgrade from 6.8.14 to 7.11.2.

These are the alerts it came with:

Any updates on this? I'm seeing the same behavior; also not using any remote cluster settings or CCR.

Also, it looks like I have 14 different 'stock' alerts that are producing the error:

CCR read exceptions
CPU usage
Cluster health
Disk Usage
Elasticsearch version mismatch
Kibana version mismatch
License expiration
Logstash version mismatch
Memory Usage (JVM)
Missing monitoring data
Nodes changed
Shard size
Thread pool search rejections
Thread pool write rejections

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.