Kibana no longer working after upgrade

Elastic Stack Kibana
1

Hello I am newbie to Elasticsearch / Kibana and Debian. The system was installed / maintained by a former colleague who has left our company.
I am following this procedure to upgrade to version 8.14.0.
Elasticsearch installation worked fine but Kibana doesn't start now.
When I run this command "sudo systemctl status kibana", I am getting this output :

root@luxnef101:~# sudo systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-06-04 14:58:54 CEST; 1h 4min ago
Docs: https://www.elastic.co
Process: 2283197 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid --deprecation.skip_deprecated_settings[0]=logging.dest (code=exited, status=1/FAILURE)
Main PID: 2283197 (code=exited, status=1/FAILURE)
CPU: 6.371s

Jun 04 14:58:54 luxnef101 systemd[1]: Stopped Kibana.
Jun 04 14:58:54 luxnef101 systemd[1]: kibana.service: Consumed 6.371s CPU time.
Jun 04 14:58:54 luxnef101 systemd[1]: kibana.service: Start request repeated too quickly.
Jun 04 14:58:54 luxnef101 systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 04 14:58:54 luxnef101 systemd[1]: Failed to start Kibana.
root@luxnef101:~#

When I open /var/log/kibana/kibana.log with nano, the file is empty.

journalctl -u kibana :

Jun 04 16:43:33 luxnef101 kibana[3731]:  FATAL  Error: [config validation of [logging].policy]: definition for this key is missing
Jun 04 16:43:33 luxnef101 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jun 04 16:43:33 luxnef101 systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 04 16:43:33 luxnef101 systemd[1]: kibana.service: Consumed 6.096s CPU time.
Jun 04 16:43:36 luxnef101 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Jun 04 16:43:36 luxnef101 systemd[1]: Stopped Kibana.
Jun 04 16:43:36 luxnef101 systemd[1]: kibana.service: Consumed 6.096s CPU time.
Jun 04 16:43:36 luxnef101 systemd[1]: Started Kibana.
Jun 04 16:43:36 luxnef101 kibana[3751]: Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.18/production.html#openssl-legacy-p>
Jun 04 16:43:38 luxnef101 kibana[3751]: {"log.level":"info","@timestamp":"2025-06-04T14:43:38.267Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.13.0","env":{"pid":3751,"proctitle":"/usr/share/kibana/bin/..>
Jun 04 16:43:38 luxnef101 kibana[3751]: Native global console methods have been overridden in production environment.
Jun 04 16:43:41 luxnef101 kibana[3751]: Configuring logger failed: ValidationError: [config validation of [logging].policy]: definition for this key is missing
Jun 04 16:43:41 luxnef101 kibana[3751]:     at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:124:13)
Jun 04 16:43:41 luxnef101 kibana[3751]:     at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/src/config_service.js:298:19)
Jun 04 16:43:41 luxnef101 kibana[3751]:     ... 40 lines matching cause stack trace ...
Jun 04 16:43:41 luxnef101 kibana[3751]:     at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:41:25)
Jun 04 16:43:41 luxnef101 kibana[3751]:     at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:123:47)
Jun 04 16:43:41 luxnef101 kibana[3751]:     at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:35:31
Jun 04 16:43:41 luxnef101 kibana[3751]:     at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Jun 04 16:43:41 luxnef101 kibana[3751]:     at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:     at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/share.js:65:18
Jun 04 16:43:41 luxnef101 kibana[3751]:     at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Jun 04 16:43:41 luxnef101 kibana[3751]:     at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30 {
Jun 04 16:43:41 luxnef101 kibana[3751]:   cause: SchemaTypeError: definition for this key is missing
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ObjectType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:165:14)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.error (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:100:47)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.internals.finalize (/usr/share/kibana/node_modules/joi/lib/validator.js:572:36)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:433:38)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.internals.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:242:28)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.exports.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:27:30)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at internals.Base.validate (/usr/share/kibana/node_modules/joi/lib/base.js:548:26)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:116:29)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/src/config_service.js:298:19)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/@kbn/config/src/config_service.js:311:189
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:10:37
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/distinctUntilChanged.js:18:28
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:10:24
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/ReplaySubject.js:54:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:41:25)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:123:47)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:35:31
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/share.js:65:18
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:9:16
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/distinctUntilChanged.js:13:16
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:9:16
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/switchMap.js:17:63
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/ReplaySubject.js:54:24)
Jun 04 16:43:41 luxnef101 kibana[3751]:       at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:41:25) {
Jun 04 16:43:41 luxnef101 kibana[3751]:     cause: undefined,
Jun 04 16:43:41 luxnef101 kibana[3751]:     path: [ 'policy' ]
Jun 04 16:43:41 luxnef101 kibana[3751]:   }
Jun 04 16:43:41 luxnef101 kibana[3751]: }
Jun 04 16:43:41 luxnef101 kibana[3751]:  FATAL  Error: [config validation of [logging].policy]: definition for this key is missing
Jun 04 16:43:41 luxnef101 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jun 04 16:43:41 luxnef101 systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 04 16:43:41 luxnef101 systemd[1]: kibana.service: Consumed 5.882s CPU time.
Jun 04 16:43:44 luxnef101 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Jun 04 16:43:44 luxnef101 systemd[1]: Stopped Kibana.
Jun 04 16:43:44 luxnef101 systemd[1]: kibana.service: Consumed 5.882s CPU time.
Jun 04 16:43:44 luxnef101 systemd[1]: kibana.service: Start request repeated too quickly.
Jun 04 16:43:44 luxnef101 systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 04 16:43:44 luxnef101 systemd[1]: Failed to start Kibana.
lines 4233-4272/4272 (END)

How can I check the system logs to figure out what is happening with Kibano ?

Any help would be appreciated.

2

Upgrade from which version? You didn't mention.

3

Hello Leandro, from version 7.17.7

4

Leandro, I see this error in the journactl :

Jun 05 10:11:45 luxnef101 kibana[187953]: Configuring logger failed: ValidationError: [config validation of [logging].policy]: definition for this key is missing

5

Please share your entire kibana.yml.

6

Sure here it is

telemetry.enables: false
telemetry.optIn: false
newsfeed.enabled: false

server.port: 5601
server.host: '172.25.0.13'
server.name: "test.com"
server.publicBaseUrl: 'https://192.168.1.13:5601'
server.maxPayload: 8388608

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/test/test.crt
server.ssl.key: /etc/kibana/certs/test/test.key
server.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt

elasticsearch.hosts: ['https://192.68.1.13:9200']

elasticsearch.username: 'kibana_system'
elasticsearch.password: 'xxx'
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt
elasticsearch.ssl.key: /etc/kibana/certs/test/test.key
elasticsearch.ssl.certificate: /etc/kibana/certs/test/test.crt
elasticsearch.ssl.verificationMode: 'certificate'
elasticsearch.requestTimeout: 132000
elasticsearch.shardTimeout: 120000

logging.root.level: debug

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
         type: pattern
  root:
    appenders:
      - default
      - file

pid.file: /run/kibana/kibana.pid

kibana.autocompleteTimeout: 2000
kibana.autocompleteTerminateAfter: 500000

monitoring.enabled: true
monitoring.kibana.collection.enabled: true
monitoring.kibana.collection.interval: 30000

monitoring.ui.enabled: true
monitoring.ui.min_interval_seconds: 20

xpack.maps.showMapVisualizationTypes: true

xpack.security.enabled: true
xpack.security.audit.enabled: false

xpack.encryptedSavedObjects.encryptionKey: 'ElastiFlow_0123456789_0123456789_0123456789'
7

I do not see anything immediately wrong with your kibana.yaml, but since Kibana complains about a logging setting, I would start by commenting out the logging stuff:

# logging.root.level: debug

# logging:
#   appenders:
#     file:
#       type: file
#       fileName: /var/log/kibana/kibana.log
#       layout:
#          type: pattern
#   root:
#     appenders:
#       - default
#       - file

and see if Kibana now starts, especially as this document, Logging settings, says that:

You do not need to configure any additional settings to use the logging features in Kibana. Logging is enabled by default and will log at info level using the pattern layout, which outputs logs to stdout .

8

Hello Jan, thank you for your feedback.
After making this change kibana is starting but it gives now this error :

root@luxnef101:~# sudo systemctl status kibana
● kibana.service - Kibana
     Loaded: loaded (/lib/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2025-06-06 10:14:34 CEST; 5min ago
       Docs: https://www.elastic.co
   Main PID: 239935 (node)
      Tasks: 11 (limit: 19140)
     Memory: 462.0M
        CPU: 1min 10.847s
     CGroup: /system.slice/kibana.service
             └─239935 /usr/share/kibana/bin/../node/glibc-217/bin/node /usr/share/kibana/bin/../src/cli/dist

Jun 06 10:15:11 luxnef101 kibana[239935]: [2025-06-06T10:15:11.891+02:00][INFO ][plugins.alerting] using indexes and aliases for persisting alerts
Jun 06 10:15:19 luxnef101 kibana[239935]: [2025-06-06T10:15:19.050+02:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpac>
Jun 06 10:15:19 luxnef101 kibana[239935]: [2025-06-06T10:15:19.051+02:00][INFO ][plugins.reporting.config] Hashed 'xpack.reporting.encryptionKey' for this instance: SNp7XVoOuTJ6pIGyHM6tFfd6RGRFgYS3pZSdvcR4Fso=
Jun 06 10:15:23 luxnef101 kibana[239935]: [2025-06-06T10:15:23.907+02:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
Jun 06 10:15:28 luxnef101 kibana[239935]: [2025-06-06T10:15:28.270+02:00][INFO ][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] Registering endpoint:user-artifact-packager task with timeout of [20m], interval of [60s] a>
Jun 06 10:15:28 luxnef101 kibana[239935]: [2025-06-06T10:15:28.272+02:00][INFO ][plugins.securitySolution.endpoint:complete-external-response-actions] Registering task [endpoint:complete-external-response-actions] with timeout of [5m] a>
Jun 06 10:15:35 luxnef101 kibana[239935]: [2025-06-06T10:15:35.202+02:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
Jun 06 10:15:35 luxnef101 kibana[239935]:         Root causes:
Jun 06 10:15:35 luxnef101 kibana[239935]:                 security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
Jun 06 10:15:35 luxnef101 kibana[239935]: [2025-06-06T10:15:35.829+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_s>

root@luxnef101:~#
9

You have issue with the kibana_system user either password is not OK. You can use curl to test. Maaybe ES is not up, check ES cluster log also and use curl again to validate ES access.