Kibana server is not ready yet 8.5

theacodes · November 25, 2022, 5:36am

I was using ELK 8.4 and everything was working fine until I saw pending updates on the server which included Elastic and Kibana 8.5 updates.

AND after upgrading both services are in active running status but kibana giving this error

Kibana server is not ready yet

I'm clueless, please help
@warkolm where you at sir?

Rios · November 25, 2022, 5:43am

Check kibana.log

theacodes · November 25, 2022, 5:45am

kibana.log

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:39:35.557+00:00","message":"[.kibana] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_8.5.2_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.5/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 6 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:39:35.558+00:00","message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 107570ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:41:57.482+00:00","message":"[.kibana_task_manager] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.5.2_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.5/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 7 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:41:57.482+00:00","message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 141928ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:41:57.486+00:00","message":"[.kibana] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_8.5.2_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.5/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 7 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:41:57.486+00:00","message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 141929ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:44:05.906+00:00","message":"[.kibana_task_manager] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.5.2_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.5/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 8 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:44:05.907+00:00","message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 128424ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:44:05.909+00:00","message":"[.kibana] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_8.5.2_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.5/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 8 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-25T05:44:05.909+00:00","message":"[.kibana] CLONE_TEMP_TO_TARGET -> CLONE_TEMP_TO_TARGET. took: 128423ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":379240},"span":{"id":"7452fb00be3a3b05"},"trace":{"id":"0b622131af55e74c98b30aad9dac10aa"}}

Rios · November 25, 2022, 5:55am

Timeout waiting for the status

Do you have enough disk space? Is your ES stable?

theacodes · November 25, 2022, 6:01am

When last time ELK was running properly, disk storage used was 91%
and then IT updated the server and since yesterday I'm unable to start ELK

elastic 9200 shows this

  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "*******",
  "version" : {
    "number" : "8.5.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "*********",
    "build_date" : "2022-11-17T18:56:17.538630285Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}```


and kibana url shows


Kibana server is not ready yet.

Rios · November 25, 2022, 9:17am

Not enough disk space. Keep it below 85%

theacodes · November 25, 2022, 10:06am

Yea I did some R and D and deleted some old indices via post request and freed some storage which actually started the kibana UI and now when I'm trying to access the fleet section I'm getting this error

except this all other tabs are working and logs are also coming via elastic agent

warkolm · November 29, 2022, 1:05am

Please don't @ people that aren't already helping

system · December 27, 2022, 1:06am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.