# New ELK 9.x cluster and Kibana HA **URL:** \ **Category:** Kibana\ **Created:** [December 1, 2025, 12:19pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780 "2025-12-01T12:19:26Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [December 1, 2025, 12:19pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/1 "2025-12-01T12:19:26Z")
Dears, I need your advice on installing a new ELK 9.x cluster. The cluster will be set up on VMs in the following configuration: 3 x MASTER node 8 x DATA node 2 x Kibana node 2 x Logstash node If I configure two separate Kibana nodes, these are two completely different VMs, do I need to pay special attention to any parameters in the /etc/kibana/kibana.yml file? Of course I am aware of the basic parameters such as: server.name server.hostserver.publicBaseUrl server.ssl.certificate server.ssl.key Is there anything else I should be aware of? Don't they clash with each other? I understand that the uuid will automatically be different. Thanks Best Regards, Dan ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 1, 2025, 12:34pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/2 "2025-12-01T12:34:19Z")
> [@d.silwon](#): > > If I configure two separate Kibana nodes Why (specifically) would you be doing this? Not that it’s wrong, what is _your_ reason. There will be some differences if you intend your 2x kibana instances to be behind a load balancer? Or do you rather intend just 2 independent instances kibana1 & kibana2, if so why? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 1, 2025, 12:53pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/3 "2025-12-01T12:53:24Z")
> [@d.silwon](#): > > Is there anything else I should be aware of? > Don't they clash with each other? There are some settings that needs to be exactly the same. Have you checked the [documentation](https://www.elastic.co/docs/deploy-manage/production-guidance/kibana-load-balance-traffic)? There is an explanation of the settings that needs to be unique and the ones that needs to be the same. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [December 1, 2025, 1:30pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/5 "2025-12-01T13:30:15Z")
@RainTown In short these will be two instances behind the LB, which ensures even distribution of traffic and in the event of a failure of one DC we need to have access to Kibana. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [December 1, 2025, 1:32pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/6 "2025-12-01T13:32:42Z")
@leandrojmp Yes, I saw this documentation. The parameter setting is not clear to me: ```auto logging: appenders: default: type: file fileName: /unique/path/per/instance ``` What in case of two separate instances/vms? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 1, 2025, 1:38pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/7 "2025-12-01T13:38:08Z")
> [@d.silwon](#): > > What in case of two separate instances/vms? Then it can be the same, the example for some reason is based on running multiple instances on the same VM, if you have different VMs, then the path can be the same, it makes no difference as the instances will not impact on each other. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [December 2, 2025, 6:30am UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/8 "2025-12-02T06:30:09Z")
How should be set up the parameter elasticsearch.hosts in /etc/kibana/kibana.yml? Does IP/hostname of nodes with role MASTER or DATA or both? In my case there are 2 x MASTER and 1 x MASTER VOTING\_ONLY ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 2, 2025, 1:18pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/9 "2025-12-02T13:18:27Z")
This should be the same in both, in your case you need to point to both your master nodes. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [December 15, 2025, 11:02am UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/10 "2025-12-15T11:02:16Z")
In Kibana logs there are some errors like this: "does not have the remote cluster client role enabled". Does this mean that I need to assign an additional role "remote\_cluster\_client" to the MASTER nodes? We plan to make index access based on roles associated with AD groups. Which nodes and with what role should have realms configuration: ```auto xpack: security: authc: realms: active_directory: my_ad: order: 0 domain_name: domain.com url: ldaps://domain.com:636 bind_dn: elastic@domain.com ssl: certificate_authorities: ["/etc/elasticsearch/domain_CA.crt"] verification_mode: certificate native: native1: order: 1 ``` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [December 15, 2025, 11:09am UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/11 "2025-12-15T11:09:47Z")
> [@d.silwon](#): > > We plan to make index access based on roles associated with AD groups. Which nodes and with what role should have realms configuration: What license do you have ? ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [December 15, 2025, 11:23am UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/12 "2025-12-15T11:23:17Z")
Platinium ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 15, 2025, 12:17pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/13 "2025-12-15T12:17:07Z")
> [@d.silwon](#): > > In Kibana logs there are some errors like this: "does not have the remote cluster client role enabled". Can you share the logs? Is this an error or a warning? > [@d.silwon](#): > > We plan to make index access based on roles associated with AD groups. Which nodes and with what role should have realms configuration You need to add this configuration in **all** nodes that Kibana is configured to make requests or else you will have issues. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [December 15, 2025, 12:46pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/14 "2025-12-15T12:46:59Z")
Sure, here you are: ```auto {"tags":["fleet-default-elastic_agent-elastic-agent-high-pipeline-queue",".es-query","rule-run-failed","framework-error"],"error":{"stack_trace":"ResponseError: illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: node [master01] does not have the remote cluster client role enabled\n at KibanaTransport._request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:533:27)\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:646:20)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/lib/wrap_scoped_cluster_client.js:132:24)\n at fetchEsqlQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_esql_query.js:44:16)\n at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:98:24)\n at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:184:14)\n at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:95:26\n at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:49:20)\n at RuleTypeRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:73:9)\n at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:236:9)\n at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:548:46)\n at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:351:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"9.0.0"},"@timestamp":"2025-12-15T10:28:19.298+01:00","message":"Executing Rule default:.es-query:fleet-default-elastic_agent-elastic-agent-high-pipeline-queue has resulted in Error: illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: node [master01] does not have the remote cluster client role enabled, caused by: \"\" - ResponseError: illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: node [master01] does not have the remote cluster client role enabled\n at KibanaTransport._request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:533:27)\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:646:20)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/lib/wrap_scoped_cluster_client.js:132:24)\n at fetchEsqlQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_esql_query.js:44:16)\n at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:98:24)\n at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:184:14)\n at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:95:26\n at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:49:20)\n at RuleTypeRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:73:9)\n at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:236:9)\n at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:548:46)\n at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:351:22)","log":{"level":"ERROR","logger":"plugins.alerting.es-query"},"process":{"pid":121064,"uptime":146.732493221},"trace":{"id":"2f7092fb51eff80173a31beb9fd0aaf4"},"transaction":{"id":"19b592319e6ce41c"}} ``` I’m still not sure if Kibana should connect to node with MASTER role. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 15, 2025, 1:19pm UTC](https://discuss.elastic.co/t/new-elk-9-x-cluster-and-kibana-ha/383780/15 "2025-12-15T13:19:19Z")
This is related to a query by a builtin-alert, in this case an observability alert: > [Elastic Agent] High pipeline queue It makes queries against remote cluster, if they exist, and it is enabled per default. You can just disable it if you do not need, just go o Observability \> Alerts and disable the ones you do not need.