When I connect via a web browser I get the error ‘Kibana server is not ready’ I’ve configured elastic with Windows certificates, I can connect to the elk cluster OK that looks OK.
https://elkcluster9200/_cluster/health
{"cluster_name":"elkprod","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":1,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
I'm using 8.11 kibana and elasticsearch running on RHEL 8
##Elasticsearch yml
cluster.name: elkprod
node.name: elkservernamenode1
path.data: /data/elasticsearch
path.logs: /app/logs/elasticsearch
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: "certs/elkprod.cer"
xpack.security.http.ssl.key: "certs/elkprod.key"
xpack.security.transport.ssl:
enabled: true
verification_mode: "certificate"
certificate: "certs/elkprod.cer"
key: "certs/elkprod.key"
certificate_authorities: [ "certs/CA_Cluster.cer", "certs/Root_CA.cer" ]
xpack.security.http.ssl.certificate_authorities: [ "certs/CA_Cluster.cer", "certs/Root_CA.cer" ]
http.host: 0.0.0.0
transport.host: 0.0.0.0
discovery.seed_hosts: [ "IP.216","IP.218","IP.219" ]
cluster.initial_master_nodes: ["elkservernamenode1","elkservernamenode2","elkservernamenode3"]
## Kibana.yml
server.host: "10.108.208.216"
elasticsearch.hosts: [ "IP.216:9200","IP.218:9200","IP.219:9200" ]
server.ssl.enabled: true
server.ssl.certificate: "/etc/kibana/config/certs/elkprod.cer"
server.ssl.key: "/etc/kibana/config/certs/elkprod.key"
xpack.security.encryptionKey: "XpZmOPWdAKVhvtvjsNPWtHvHHRjhTORV"
xpack.reporting.encryptionKey: "iiz36Zz7OqqZ8f7kM2yIrjEMgJifDTzW"
xpack.encryptedSavedObjects.encryptionKey: "vQECNynvftcYAx5z9AeQ3aNtZoiujplE"
elasticsearch.ssl.verificationMode: none
elasticsearch.serviceAccountToken: 32Token
elasticsearch.ssl.certificate: "/etc/kibana/config/certs/elkprod.cer"
elasticsearch.ssl.key: "/etc/kibana/config/certs/elkprod.key"
Both these commands seem to work:
curl -k -H "Authorization: Bearer AAEAAWVsYXN0aWMva2liYW5hL2tpYmFuYV90b2tlbi0xOk51WGNnUUxnUlVHckhYNTRlc2RXQlE" https://10.211.55.7:9200/_cluster/health
curl -H "Authorization: Bearer AAEAAWVsYXN0aWMva2liYW5hL2tpYmFuYV90b2tlbi0xOk51WGNnUUxnUlVHckhYNTRlc2RXQlE" https://10.211.55.7:9200/_cluster/health
Kibana log
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.233+00:00][DEBUG][elasticsearch.query.data] 401 - 511.0B Jan 10 15:38:39 elkservernamenode1 kibana[703628]: GET /_cluster/settings?include_defaults=true&flat_settings=true [security_exception]: failed to authenticate service account [elas> Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.233+00:00][INFO ][root] Kibana is shutting down Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.236+00:00][FATAL][root] Reason: security_exception
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: Root causes:
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: security_exception: failed to authenticate service account [elastic/kibana] with token name [elkprod]
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: ResponseError: security_exception
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: Root causes:
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: security_exception: failed to authenticate service account [elastic/kibana] with token name [elkprod]
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at KibanaTransport.request ([/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27](mailto:/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:479:27))
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport>
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at Cluster.getSettings ([/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/cluster.js:157:16](mailto:/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/cluster.js:157:16))
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at isInlineScriptingEnabled ([/usr/share/kibana/node_modules/@kbn/core-elasticsearch-server-internal/src/is_scripting_enabled.j](mailto:/usr/share/kibana/node_modules/@kbn/core-elasticsearch-server-internal/src/is_scripting_enabled.j)>
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at ElasticsearchService.start (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-server-internal/src/elasticsearch_servic>
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at Server.start ([/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/server.js:342:32](mailto:/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/server.js:342:32))
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at Root.start ([/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/root/index.js:65:14](mailto:/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/root/index.js:65:14))
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at bootstrap ([/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:115:5](mailto:/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:115:5))
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:211:5)
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.236+00:00][DEBUG][server] stopping server Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.237+00:00][DEBUG][http.server.Preboot] stopping http server
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.261+00:00][DEBUG][plugins.eventLog] Initialization failed, not indexing 1 documents Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.261+00:00][ERROR][plugins.ruleRegistry] Error: Server is stopping; must stop all async operations
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at [/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:40:18](mailto:/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:40:18)
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.262+00:00][ERROR][plugins.ruleRegistry] Error: Failure during installation of common resources shared between>
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at installWithTimeout ([/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:4](mailto:/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:4)>
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: at ResourceInstaller.installCommonResources (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_>
Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.262+00:00][DEBUG][plugins.eventLog] shutdown: finished Jan 10 15:38:39 elkservernamenode1 kibana[703628]: [2024-01-10T15:38:39.263+00:00][DEBUG][plugins-system.standard] Stopping plugin "spaces"...