New ELK 9.x cluster and Kibana HA

Elastic Stack Kibana
1

Dears,

I need your advice on installing a new ELK 9.x cluster. The cluster will be set up on VMs in the following configuration:
3 x MASTER node
8 x DATA node
2 x Kibana node
2 x Logstash node

If I configure two separate Kibana nodes, these are two completely different VMs, do I need to pay special attention to any parameters in the /etc/kibana/kibana.yml file?
Of course I am aware of the basic parameters such as:
server.name
server.hostserver.publicBaseUrl
server.ssl.certificate
server.ssl.key

Is there anything else I should be aware of?
Don't they clash with each other?

I understand that the uuid will automatically be different.

Thanks

Best Regards,
Dan

2

Why (specifically) would you be doing this? Not that it’s wrong, what is your reason.

There will be some differences if you intend your 2x kibana instances to be behind a load balancer? Or do you rather intend just 2 independent instances kibana1 & kibana2, if so why?

3

There are some settings that needs to be exactly the same.

Have you checked the documentation?

There is an explanation of the settings that needs to be unique and the ones that needs to be the same.

5

@RainTown In short these will be two instances behind the LB, which ensures even distribution of traffic and in the event of a failure of one DC we need to have access to Kibana.

1 Like
6

@leandrojmp Yes, I saw this documentation. The parameter setting is not clear to me:

logging:
  appenders:
    default:
      type: file
      fileName: /unique/path/per/instance

What in case of two separate instances/vms?

7

Then it can be the same, the example for some reason is based on running multiple instances on the same VM, if you have different VMs, then the path can be the same, it makes no difference as the instances will not impact on each other.

1 Like
8

How should be set up the parameter elasticsearch.hosts in /etc/kibana/kibana.yml? Does IP/hostname of nodes with role MASTER or DATA or both?

In my case there are 2 x MASTER and 1 x MASTER VOTING_ONLY

9

This should be the same in both, in your case you need to point to both your master nodes.

1 Like
10

In Kibana logs there are some errors like this: "does not have the remote cluster client role enabled". Does this mean that I need to assign an additional role "remote_cluster_client" to the MASTER nodes?

We plan to make index access based on roles associated with AD groups. Which nodes and with what role should have realms configuration:

xpack:
  security:
    authc:
      realms:
        active_directory:
          my_ad:
            order: 0
            domain_name: domain.com
            url: ldaps://domain.com:636
            bind_dn: elastic@domain.com
            ssl:
              certificate_authorities: [ "/etc/elasticsearch/domain_CA.crt" ]
              verification_mode: certificate
        native:
          native1:
            order: 1
11

What license do you have ?

12

Platinium

1 Like
13

Can you share the logs? Is this an error or a warning?

You need to add this configuration in all nodes that Kibana is configured to make requests or else you will have issues.

14

Sure, here you are:

{"tags":["fleet-default-elastic_agent-elastic-agent-high-pipeline-queue",".es-query","rule-run-failed","framework-error"],"error":{"stack_trace":"ResponseError: illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: node [master01] does not have the remote cluster client role enabled\n    at KibanaTransport._request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:533:27)\n    at processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:646:20)\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/lib/wrap_scoped_cluster_client.js:132:24)\n    at fetchEsqlQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_esql_query.js:44:16)\n    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:98:24)\n    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:184:14)\n    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:95:26\n    at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:49:20)\n    at RuleTypeRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:73:9)\n    at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:236:9)\n    at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:548:46)\n    at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:351:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"9.0.0"},"@timestamp":"2025-12-15T10:28:19.298+01:00","message":"Executing Rule default:.es-query:fleet-default-elastic_agent-elastic-agent-high-pipeline-queue has resulted in Error: illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: node [master01] does not have the remote cluster client role enabled, caused by: \"\" - ResponseError: illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: node [master01] does not have the remote cluster client role enabled\n    at KibanaTransport._request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:533:27)\n    at processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:646:20)\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/lib/wrap_scoped_cluster_client.js:132:24)\n    at fetchEsqlQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_esql_query.js:44:16)\n    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:98:24)\n    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:184:14)\n    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:95:26\n    at TaskRunnerTimer.runWithTimer (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner_timer.js:49:20)\n    at RuleTypeRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/rule_type_runner.js:73:9)\n    at TaskRunner.runRule (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:236:9)\n    at TaskRunner.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:548:46)\n    at TaskManagerRunner.run (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:351:22)","log":{"level":"ERROR","logger":"plugins.alerting.es-query"},"process":{"pid":121064,"uptime":146.732493221},"trace":{"id":"2f7092fb51eff80173a31beb9fd0aaf4"},"transaction":{"id":"19b592319e6ce41c"}}

I’m still not sure if Kibana should connect to node with MASTER role.

15

This is related to a query by a builtin-alert, in this case an observability alert:

[Elastic Agent] High pipeline queue

It makes queries against remote cluster, if they exist, and it is enabled per default.

You can just disable it if you do not need, just go o Observability > Alerts and disable the ones you do not need.

1 Like