Kibana failed to start (after upgrade 7.4 > 7.16)

Elastic Stack Kibana
1

Hello,

After updating my Elasticsearch nodes from 7.4 to 7.16, I followed the upgrade documentation and upgraded my Kibana node from 7.4 to 7.16 through rpm (rpm -Uvh kibana[...].rpm

The install process finished and everything seems ok
I have a warning for the creation of a /etc/kibana/kibana.yml.rpmsave file so I replaced the old kibana.yml with the .rpmsave one, carefully replacing each needed field with the right values (node.name, elastic hosts, ...)

Then when I try to start the kibana service through systemctl start kibana, kibana starts and then fail (about 5-10s after).

The end of the kibana.log (logging.dest has been defined in the kibana.yml) shows this message.

{"type":"log","@timestamp":"2021-12-30T15:39:58+01:00","tags":["warning","config","deprecation"],"pid":19973,"message":"Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout."}
{"type":"log","@timestamp":"2021-12-30T15:39:59+01:00","tags":["fatal","root"],"pid":19973,"message":"Error: Connection with id 'http://10.118.124.232:8082/' is already present\n    at ConnectionPool.addConnection (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/pool/BaseConnectionPool.js:109:13)\n    at /usr/share/kibana/node_modules/@elastic/elasticsearch/lib/pool/BaseConnectionPool.js:98:37\n    at Array.forEach (<anonymous>)\n    at ConnectionPool.addConnection (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/pool/BaseConnectionPool.js:98:19)\n    at new Client (/usr/share/kibana/node_modules/@elastic/elasticsearch/index.js:171:27)\n    at configureClient (/usr/share/kibana/src/core/server/elasticsearch/client/configure_client.js:46:18)\n    at new ClusterClient (/usr/share/kibana/src/core/server/elasticsearch/client/cluster_client.js:48:65)\n    at ElasticsearchService.createClusterClient (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:135:12)\n    at ElasticsearchService.setup (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:72:24)\n    at Server.setup (/usr/share/kibana/src/core/server/server.js:251:39)\n    at Root.setup (/usr/share/kibana/src/core/server/root/index.js:58:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:119:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:229:5)"}

Can you help me ?

Thank you

2

Are you running Kibana on port 8082? Can you check which process is running on port 8082?

3

Hello,
No I am not running Kibana on port 8082.
The listen.port is still 5601 for KIbana.

8082 stands for the listen port of my Elasticsearch (master) nodes (they are all on different hosts)
I tried to stop my 3 master nodes and restart them, but I got no more success.

Update of the Elasticsearch nodes is done and ok in version 7.16.
That's only when I updated Kibana the problem appeared.

EDIT:
If needed I put a diff of my original 7.4 "kibana.yml_bkp_..." vs the new "kibana.yml" issued from the 7.16 install.

[kibana]# diff kibana.yml_7.4 kibana.yml
20a21,24
> # Specifies the public URL at which Kibana is available for end users. If
> # `server.basePath` is configured this URL should end with the same basePath.
> #server.publicBaseUrl: ""
>
22c26
< #server.maxPayloadBytes: 1048576
---
> #server.maxPayload: 1048576
30,34d33
< # When this setting's value is true Kibana uses the hostname specified in the server.host
< # setting. When the value of this setting is false, Kibana uses the hostname of the host
< # that connects to this Kibana instance.
< #elasticsearch.preserveHost: true
<
48a48,51
> # Kibana can also authenticate to Elasticsearch via "service account tokens".
> # If may use this token instead of a username/password.
> # elasticsearch.serviceAccountToken: "my_token"
>
56c59,60
< # These files validate that your Elasticsearch backend uses the same key files.
---
> # These files are used to verify the identity of Kibana to Elasticsearch and are required when
> # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
86,88d89
< # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
< #elasticsearch.startupTimeout: 5000
<
93c94
< #pid.file: /usr/share/kibana/kibana.pid
---
> #pid.file: /run/kibana/kibana.pid
95c96
< # Enables you specify a file where Kibana stores log output.
---
> # Enables you to specify a file where Kibana stores log output.
99c100
< #logging.silent: false
---
> logging.silent: false
4

Problem solved, sorry for bothering.

I tried to remove the "problematic" elastic host from kibana.yml file and then started the kibana node.
It went ok....
I then restarted all my nodes and re-add the last Elasticsearch node in the kibana.yml.
Kibana booted correctly.

Not sure of what was the problem, but no more problem now.

Thank you all for trying to help.