Hi Pal, I recently encountered an issue from my Kibana 8.11.1 version upgrade. The token I used is from monitoring node, this is for the kibana and monitoring connections.
However, the kibana UI shows "Kibana server is not ready yet." As per kibana logs:
[2024-01-07T04:48:44.506+11:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
Root causes:
security_exception: failed to authenticate service account [elastic/kibana] with token name [token_Tw3W34wBTG72srMePoYC]
[2024-01-07T04:48:44.506+11:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
Root causes:
security_exception: failed to authenticate service account [elastic/kibana] with token name [token_Tw3W34wBTG72srMePoYC]
[2024-01-07T04:48:45.202+11:00][INFO ][plugins.screenshotting.chromium] Browser executable: /instace/kibana-8.11.1/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
[2024-01-07T04:48:45.202+11:00][INFO ][plugins.screenshotting.chromium] Browser executable: /instace/kibana-8.11.1/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
[2024-01-07T05:08:43.661+11:00][ERROR][plugins.ruleRegistry] Error: Timeout: it took more than 1200000ms
at Timeout._onTimeout (/instace/kibana-8.11.1/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:36:18)
at listOnTimeout (node:internal/timers:569:17)
at processTimers (node:internal/timers:512:7)
[2024-01-07T05:08:43.661+11:00][ERROR][plugins.ruleRegistry] Error: Timeout: it took more than 1200000ms
at Timeout._onTimeout (/instace/kibana-8.11.1/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:36:18)
at listOnTimeout (node:internal/timers:569:17)
at processTimers (node:internal/timers:512:7)
[2024-01-07T05:08:43.663+11:00][ERROR][plugins.ruleRegistry] Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms
at installWithTimeout (/instace/kibana-8.11.1/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:48:11)
at ResourceInstaller.installCommonResources (/instace/kibana-8.11.1/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)
[2024-01-07T05:08:43.663+11:00][ERROR][plugins.ruleRegistry] Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms
at installWithTimeout (/instace/kibana-8.11.1/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:48:11)
at ResourceInstaller.installCommonResources (/instace/kibana-8.11.1/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)
Here is the kibana.yml:
server.port: 5602
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: 0.0.0.0
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
server.name: ${HOSTNAME}
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://elastic01.example.com:9202", "https://elastic02.example.com:9202", "https://elastic03.example.com:9202", "https://elastic04.example.com:9202"]
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"
# The default application to load.
#kibana.defaultAppId: "home"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "elastic"
#elasticsearch.password: "old_password"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false
# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid
# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
# logging.quiet: true
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: true
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
server.ssl.enabled: true
server.ssl.certificate: certs/elastic.example.com/cert1.pem
server.ssl.key: certs/kibana.example.com.key
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: ["certs/RootCertCA01.crt","certs/CertCA01.crt"]
elasticsearch.ssl.certificate: certs/elastic.example.com/cert1.pem
elasticsearch.ssl.key: certs/kibana.example.com.key
monitoring.kibana.collection.enabled: true
monitoring.ui.enabled: true
monitoring.ui.elasticsearch.hosts: ["https://monitoring:9202"]
monitoring.ui.elasticsearch.ssl.certificateAuthorities: ["certs/RootCertCA01.crt","certs/CertCA01.crt"]
monitoring.ui.elasticsearch.ssl.verificationMode: certificate
monitoring.ui.elasticsearch.ssl.certificate: certs/elastic.example.com/cert1.pem
monitoring.ui.elasticsearch.ssl.key: certs/kibana.example.com.key
#monitoring.ui.elasticsearch.username: "elastic"
#monitoring.ui.elasticsearch.password: "old_password"
monitoring.ui.elasticsearch.serviceAccountToken: "AAEAAWVsYXN0aWMva2liYW5hL3Rva2VuX1R3M1czNHdCVEc3MnNyTWVQb1lDOmUwMHNzV0NiVEVTODVhZU4xQWRLS1E"
elasticsearch.serviceAccountToken: "AAEAAWVsYXN0aWMva2liYW5hL3Rva2VuX1R3M1czNHdCVEc3MnNyTWVQb1lDOmUwMHNzV0NiVEVTODVhZU4xQWRLS1E"
logging:
root:
level: info
appenders: [rolling-file,default]
appenders:
rolling-file:
type: rolling-file
fileName: /instance/kibana-8.11.1/logs/kibana.log
policy:
type: time-interval
interval: 12h
modulate: true
strategy:
type: numeric
pattern: '-%i'
max: 10
layout:
type: pattern
For the certificate, this is the verification:
This is already resolved by adding the