KIbana server not ready yet. Update from 7.5.2 to 7.6.2

Elastic Stack Kibana
1

Hi, dear community!

I updated Kibana and Elasticsearch from version 7.5.2 to 7.6.2 and received an error - KIbana server not ready yet. I do not see anything suspicious in the logs except that the Kibana service is restarted every four minutes by unknown reason. I will very appreciate some help, thank you.

kibana log - https://pastebin.com/meBySpE9
elastic log - https://pastebin.com/vrRkVf92
netstat, nodes and cluster health screenshot: https://imgur.com/a/JdzGaX7

2

I don't see anything weird in any of the logs. If I understand the issue correctly - the Kibana server starts but it restarts itself every 4 minutes? Are you self-hosting Kibana server? Or somewhere in the cloud?

3

Hello, @majagrubic, thank you for the reply!

Yes, we're have self-hosted Kibana and Elastic. Yes, Kibana starts correctly, as far as i know, but web-interface doesn't load properly and show error.

4

What error is the interface showing?

5

1

6

Ok, can you try the following things:

  1. Check that Elasticsearch instance is running correctly curl -X GET "localhost:9200/_cluster/health?pretty"
  2. Check that you accidentally don't have a previous Kibana instance running
  3. Verify that kibana is not trying to load some plugins which are not installed on the master node
  4. Restart Kibana server
  5. If none of that doesn't help, delete .kibana* indices from Elasticsearch. Note that this will delete your Kibana data (dashboards, visualizations etc).
7
  1. Elastic cluster is yellow, but i know why, that's not depend on Kibana node. Elastic is working, i checked it.
  2. Looks like no.
  3. I have only one plugin and it loads properly, as far as i know by logs.
  4. Done, doesn't help.
  5. Done, doesn't help.

1

2
21743×115 54.6 KB

8

I found interesting thing: Kibana restarts cause to failing - system is restart it due to fail. Investigating reason.

9

I'm a bit surprised deleting the .kibana indices didn't solve the problem.

10

Hello, @majagrubic! Problem is still here, i have no idea, what to do. Maybe you do?

11

In your kibana.yml file could you please update the following setting:
logging.verbose: true (notice you'll need to uncomment it if commented out now), restart Kibana and paste the logs again?

12

Thank you for the reply.
Sure, let me check

13

Here is log with debug enabled:
https://pastebin.com/2FdhjHyQ

I see nothing suspicious.

14

Looks like it runs the optimizer, but that shouldn't fail. What happens if you remove the /optimize folder from the Kibana install dir? Maybe something got corrupted there and it messing with the optimizer service.

15

Hello, @Marius_Dragomir, thank you for the reply.
Now it failed with the next errors:

апр 16 12:20:20 kibana kibana[21025]: {"type":"log","@timestamp":"2020-04-16T12:20:20Z","tags":["fatal","root"],"pid":21025,"message":"{ [Error: ENOENT: no such file or directory, mkdir] errno: -2, code: 'ENOENT', syscall: 'mkdir' }"}

апр 16 12:20:20 kibana kibana[21025]: FATAL  Error: ENOENT: no such file or directory, mkdir
апр 16 12:20:21 kibana systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE

Then I made /optimize folder manually and set it owner to kibana. It returns to the beginning of all.

16

Maybe some permissions are wrong? I checked all refer to this manual: https://documentation.wazuh.com/3.12/upgrade-guide/upgrading-elastic-stack/elastic_server_minor_upgrade.html

Also i try clean install, doesn't help.

17

Hi again, @Marius_Dragomir, @majagrubic :slight_smile:
I have found a solution. I increased NODE_OPTIONS="--max_old_space_size=2048" in /etc/default/kibana to 4096 and started the kibana in manual mode, not as a service. Optimization took place in 4 minutes and the service started up successfully.

/usr/share/kibana/bin/kibana --optimize --allow-root --verbose -e "http://kibana_ip:9200"

Thank you for the help

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.