Failed to Start Kibana 7.16.1

Elastic Stack Kibana
1

After Upgrading from 7.5.2or with fresh installation of Kibana 7.16.1 - it is failed to start.
7.5.2 version was working well.
ROR plugin is used for authentication. This is also upgraded to readonlyrest_kbn_enterprise-1.37.0_es7.16.1

systemctl status -l kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Mon 2022-01-10 08:41:14 UTC; 28s ago
     Docs: https://www.elastic.co
  Process: 3336 ExecStart=/usr/share/kibana/bin/kibana --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid" (code=exited, status=1/FAILURE)
 Main PID: 3336 (code=exited, status=1/FAILURE)

Jan 10 08:41:11 dev-elk-kb-7a-1.novalocal systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Jan 10 08:41:11 dev-elk-kb-7a-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Jan 10 08:41:11 dev-elk-kb-7a-1.novalocal systemd[1]: kibana.service failed.
Jan 10 08:41:14 dev-elk-kb-7a-1.novalocal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jan 10 08:41:14 dev-elk-kb-7a-1.novalocal systemd[1]: Stopped Kibana.
Jan 10 08:41:14 dev-elk-kb-7a-1.novalocal systemd[1]: start request repeated too quickly for kibana.service
Jan 10 08:41:14 dev-elk-kb-7a-1.novalocal systemd[1]: Failed to start Kibana.
Jan 10 08:41:14 dev-elk-kb-7a-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Jan 10 08:41:14 dev-elk-kb-7a-1.novalocal systemd[1]: kibana.service failed.
[centos@dev-elk-kb-7a-1 ~]$ timed out waiting for input: auto-logout
2

Hi, @abhayamishra. The attached logs don't look like written by Kibana. Could you run Kibana with --verbose flag and the generated logs?

3

Welcome to our community! :smiley:

You will need to look in /var/log/kibana/kibana.log, as it will have more information.

4

Ahh!!!.. Forgot to mention, there is NO logs getting generated, as soon as I do upgrade or install this version, so unable to identify issue.
Attached --verbose flag, sharing output.

[centos@dev-elk-kb-7a-1 ~]$ systemctl status -l kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2022-01-11 05:26:45 UTC; 18s ago
     Docs: https://www.elastic.co
  Process: 1097 ExecStart=/usr/share/kibana/bin/kibana --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid" --verbose (code=exited, status=1/FAILURE)
 Main PID: 1097 (code=exited, status=1/FAILURE)

Jan 11 05:26:42 dev-elk-kb-7a-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Jan 11 05:26:42 dev-elk-kb-7a-1.novalocal systemd[1]: kibana.service failed.
Jan 11 05:26:45 dev-elk-kb-7a-1.novalocal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jan 11 05:26:45 dev-elk-kb-7a-1.novalocal systemd[1]: Stopped Kibana.
Jan 11 05:26:45 dev-elk-kb-7a-1.novalocal systemd[1]: start request repeated too quickly for kibana.service
Jan 11 05:26:45 dev-elk-kb-7a-1.novalocal systemd[1]: Failed to start Kibana.
Jan 11 05:26:45 dev-elk-kb-7a-1.novalocal systemd[1]: Unit kibana.service entered failed state.
Jan 11 05:26:45 dev-elk-kb-7a-1.novalocal systemd[1]: kibana.service failed.

Last logs generated yesterday, before upgrade, after upgrade there is no logs getting generated.

{"type":"log","@timestamp":"2022-01-10T08:37:10Z","tags":["info","plugins-system"],"pid":14995,"message":"Stopping all   plugins."}
5

Please share more of the Kibana log.

6

After upgrade or fresh install of 7.16.1, there is no logs getting generated.

Should we possibly look into some configuration?

7

Let's give it a try, could you attach your config file?

Are you sure your orchestration system runs Kibana?

8

Yes, This is all working before upgrading it from 7.5.2, to 7.16.1. After upgrade to 7.16.1, server is not getting started. There are still other servers where Kibana 7.5.2 is running and no issue. We are trying to upgrade due to Log4j Vulnerability (although this is not impacted, but ES and Logstash are already upgraded to 7.16.1)

Kibana.yml

server.port: 5601
server.host: "0.0.0.0"
server.basePath: ""
server.maxPayloadBytes: 1048576
server.name: "dev-elk-kb-7b-1-10.220.xxx.xx"
elasticsearch.hosts: "https://read-es.elk-xxx.xxx.regn.net:9200"
elasticsearch.preserveHost: true
kibana.index: ".kibana"
kibana.defaultAppId: "discover"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/WC-elk-ins.cert.pem
server.ssl.key: /etc/kibana/WC-elk-ins.key.pem
server.ssl.certificateAuthorities: ["/etc/kibana/elk-ins-ca.crt"]
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/elk-ins-ca.crt"]
elasticsearch.ssl.verificationMode: none
elasticsearch.requestTimeout: 150000
elasticsearch.shardTimeout: 0
elasticsearch.startupTimeout: 5000
pid.file: /var/run/kibana/kibana.pid
logging.dest: /var/log/kibana/kibana.log
logging.quiet: true
logging.verbose: false
ops.interval: 5000
elasticsearch.username: "xxxx"
elasticsearch.password: "xxxx"
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.reporting.enabled: true
xpack.reporting.encryptionKey: "190923562656492345678901678901234545285761"
readonlyrest_kbn.login_custom_logo: 'https://xxx.xxx.com/-/media/images/logos/logo_.png'
9

I'd start by removing/disabling the read only rest plugin and seeing if this still happens.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.