Kibana server is not ready yet

Elastic Stack Kibana
1

After upgrade elk to 7.9.2:

root@log:/# systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-09-28 10:06:25 -03; 1h 45min ago
Main PID: 9089 (node)
Tasks: 11 (limit: 4915)
Memory: 1016.2M
CGroup: /system.slice/kibana.service
└─9089 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli

set 28 10:07:54 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:54Z","tags":["warning","plugins","reporting","config"],"pid":9089,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions fr
set 28 10:07:54 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:54Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":9089,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey
set 28 10:07:54 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:54Z","tags":["warning","plugins","ingestManager"],"pid":9089,"message":"Fleet APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral e
set 28 10:07:55 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:55Z","tags":["warning","plugins","actions","actions"],"pid":9089,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral enc
set 28 10:07:55 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:55Z","tags":["warning","plugins","alerts","plugins","alerting"],"pid":9089,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an eph
set 28 10:07:55 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:55Z","tags":["info","plugins","monitoring","monitoring"],"pid":9089,"message":"config sourced from: production cluster"}
set 28 10:07:55 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:55Z","tags":["info","savedobjects-service"],"pid":9089,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objec
set 28 10:07:55 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:55Z","tags":["info","savedobjects-service"],"pid":9089,"message":"Starting saved objects migrations"}
set 28 10:07:55 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:55Z","tags":["warning","plugins","reporting","config"],"pid":9089,"message":"Chromium sandbox provides an additional layer of protection, but is not supported
set 28 10:07:55 log kibana[9089]: {"type":"log","@timestamp":"2020-09-28T13:07:55Z","tags":["warning","savedobjects-service"],"pid":9089,"message":"Unable to connect to Elasticsearch. Error: [search_phase_execution_exception] all shards
lines 1-19/19 (END)

2

It looks like your Kibana cannot access Elasticsearch, can you check that part?

Best,
Oleg

3

exactly! my elk was working pretty good, but when i decided to upgrade it. It started to fail. elastic and logstsh are working, but kibana cannot connect to elastic.

Do you know if is this due to different versions of them? how can i fix it?

4

Do you know if is this due to different versions of them?

Well, Kibana and ES versions should match, see Support Matrix | Elastic. But it's hard to tell what the actual failure reason in your case is without full debug logs and configuration. Usually Kibana tells in the logs that it detected wrong ES version, but Error: [search_phase_execution_exception bit likely means the issue is somewhere else. Try to enable --logging.verbose: true and see what you can notice in the logs, feel free to share them here as well (once you remove all sensitive info from there).

Best,
Oleg

5

Hi, oleg!
Does it help? thank you!

set 28 14:15:37 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:37Z","tags":["debug","plugins","fileUpload"],"pid":16245,"message":""/usr/share/kibana/x-pack/plugins/file_upload/server" does not export "config"."}
set 28 14:15:37 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:37Z","tags":["debug","config"],"pid":16245,"message":"Marking config path as handled: xpack,eventLog"}
set 28 14:15:38 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:38Z","tags":["debug","config"],"pid":16245,"message":"Marking config path as handled: xpack,graph"}
set 28 14:15:38 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:38Z","tags":["debug","plugins","grokdebugger"],"pid":16245,"message":""/usr/share/kibana/x-pack/plugins/grokdebugger/server" does not export "config"."}
set 28 14:15:39 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:39Z","tags":["debug","config"],"pid":16245,"message":"Marking config path as handled: xpack,index_management"}
set 28 14:15:39 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:39Z","tags":["debug","plugins","globalSearchProviders"],"pid":16245,"message":""/usr/share/kibana/x-pack/plugins/global_search_providers/server" does not e
set 28 14:15:39 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:39Z","tags":["debug","plugins","ingestPipelines"],"pid":16245,"message":""/usr/share/kibana/x-pack/plugins/ingest_pipelines/server" does not export "confi
set 28 14:15:45 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:45Z","tags":["debug","config"],"pid":16245,"message":"Marking config path as handled: xpack,infra"}
set 28 14:15:45 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:45Z","tags":["debug","config"],"pid":16245,"message":"Marking config path as handled: xpack,ilm"}
set 28 14:15:46 log kibana[16245]: {"type":"log","@timestamp":"2020-09-28T17:15:46Z","tags":["debug","config"],"pid":16245,"message":"Marking config path as handled: xpack,lens"}

6

Nope, unfortunately. It just excerpt from the logs, we really need a full log, from the start of the KIbana till a couple of minutes later to make sure it tried to bootstrap and stuck and not just slowly bootstrapping.

Best,
Oleg

7

Ok, sorry! I'm new with these tools, but now i got it. here we go... I stopped ELK and then started in the following order: first elastic, kibana and logstash.

the following doc contains the logs after minutes that i started the services.

Any idea?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.