Cannot connect to Kibana(no response)

Elastic Stack Kibana
1

I installed Elasticsearch 8.16 successfully

❯ curl localhost:9200
{
  "name" : "debian",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "8bKtWy_zTImqaRYnzC453g",
  "version" : {
    "number" : "8.16.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "12ff76a92922609df4aba61a368e7adf65589749",
    "build_date" : "2024-11-08T10:05:56.292914697Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

# with secureity disabled in elasticsearch.yml
xpack.security.enabled: false
xpack.security.enrollment.enabled: false

However, I can't seem to connec to Kibana at all. It started successfull but I couldn't get a response from it.

❯ curl localhost:5601
# keep waiting and nothing happened

❯ sudo service kibana status
● kibana.service - Kibana
     Loaded: loaded (/lib/systemd/system/kibana.service; disabled; preset: enabled)
     Active: active (running) since Fri 2024-11-15 11:22:50 AEDT; 20min ago
       Docs: https://www.elastic.co
   Main PID: 4081935 (node)
      Tasks: 11 (limit: 18991)
     Memory: 418.4M
        CPU: 24.376s
     CGroup: /system.slice/kibana.service
             └─4081935 /usr/share/kibana/bin/../node/glibc-217/bin/node /usr/share/kibana/bin/../src/cli/dist

Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.203+11:00][INFO ][elasticsearch-service] Successfully connected to Elasticsearch after waiting for 1193856 milliseconds
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.296+11:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting s>
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.297+11:00][INFO ][savedobjects-service] Starting saved objects migrations
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.340+11:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 21ms.
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.343+11:00][INFO ][savedobjects-service] [.kibana_alerting_cases] INIT -> CREATE_NEW_TARGET. took: 18ms.
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.344+11:00][INFO ][savedobjects-service] [.kibana_ingest] INIT -> CREATE_NEW_TARGET. took: 17ms.
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.344+11:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 28ms.
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.347+11:00][INFO ][savedobjects-service] [.kibana_security_solution] INIT -> CREATE_NEW_TARGET. took: 24ms.
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.348+11:00][INFO ][savedobjects-service] [.kibana_analytics] INIT -> CREATE_NEW_TARGET. took: 28ms.
Nov 15 11:42:55 debian kibana[4081935]: [2024-11-15T11:42:55.349+11:00][INFO ][savedobjects-service] [.kibana_usage_counters] INIT -> CREATE_NEW_TARGET. took: 32ms.

Here is the log from /var/log/kibana/kibana.log

❯ sudo tail /var/log/kibana/kibana.log
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.967+11:00","message":"[.kibana_analytics] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_analytics_8.16.0_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.16/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 1 in 2 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.504607684},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.967+11:00","message":"[.kibana_analytics] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 600619ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.505666319},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.969+11:00","message":"[.kibana_task_manager] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.16.0_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.16/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 1 in 2 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.506383457},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.969+11:00","message":"[.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 600629ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.506484616},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.970+11:00","message":"[.kibana_ingest] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_ingest_8.16.0_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.16/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 1 in 2 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.507199469},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.970+11:00","message":"[.kibana_ingest] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 600626ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.50879695},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.972+11:00","message":"[.kibana] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_8.16.0_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.16/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 1 in 2 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.509178782},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.972+11:00","message":"[.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 600628ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.509279941},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.972+11:00","message":"[.kibana_usage_counters] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_usage_counters_8.16.0_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.16/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 1 in 2 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.510135246},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-11-15T11:52:55.973+11:00","message":"[.kibana_usage_counters] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 600623ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":4081935,"uptime":1805.510243087},"trace":{"id":"5312a1e9627913e39de360d7c9d86ab6"},"transaction":{"id":"aa84d5e48d84a657"}}
2

Hi @chung

Hmm that took a long time...

You need to share your kibana.yml...

Is this an initial install or an upgrade?

3

Initially it was an upgrade(from Elasticsearch and Kibana 7.15.2) to 8.16. Then I forced me to have an intermediate upgrade to 7.17 before I could continue to 8.16. I decided to purge everything and start fresh as I don't care about any migration at all.

apt-get purge elasticsearch kibana
apt-get install elasticsearch kibana

I believe the kibana.yml I have is the stock one as I purged everything before the installation.

# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"

# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# The maximum number of sockets that can be used for communications with elasticsearch.
# Defaults to `Infinity`.
#elasticsearch.maxSockets: 1024

# Specifies whether Kibana should use compression for communications with elasticsearch
# Defaults to `false`.
#elasticsearch.compression: false

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'
#logging.root.level: debug

# Enables you to specify a file where Kibana stores log output.
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
#  policy:
#    type: size-limit
#    size: 256mb
#  strategy:
#    type: numeric
#    max: 10
#  layout:
#    type: json

# Logs queries sent to Elasticsearch.
#logging.loggers:
#  - name: elasticsearch.query
#    level: debug

# Logs http responses.
#logging.loggers:
#  - name: http.server.response
#    level: debug

# Logs system usage information.
#logging.loggers:
#  - name: metrics.ops
#    level: debug

# Enables debug logging on the browser (dev console)
#logging.browser.root:
#  level: debug

# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data

# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000ms.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR".
#i18n.locale: "en"

# =================== Frequently used (Optional)===================

# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.

# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000 objects per batch.
#migrations.batchSize: 1000

# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
#migrations.maxBatchSizeBytes: 100mb

# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
#migrations.retryAttempts: 15

# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000ms
#unifiedSearch.autocomplete.valueSuggestions.timeout: 1000

# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100_000
#unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000
4

Did you take a look at (as recommended in the log messages)

Since you upggraded from 7.17..I would take a look at the whole page...

5

Thank you very much for your help.

I solved the problem by manually removing the related directories under
/var/lib and /usr/share following another reinstallation.

1 Like