Failed to start Kibana after upgrade 7.17 to 8.12

michael8 · March 8, 2024, 11:59am

Hi,

I've already looked through all the posts about errors and I can't find a solution for myself.

After updating Kibana from 7.17 to 8.12, Kibana does not want to start.

All errors were previously eliminated. Elastic updated to version 8.12 and works fine.

Below is what I have in the files:


kibana.sercices:
[Unit]
Description=Kibana
Documentation=https://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=kibana
Group=kibana
PrivateTmp=true

Environment=KBN_HOME=/usr/share/kibana
Environment=KBN_PATH_CONF=/etc/kibana

EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana

ExecStart=/usr/share/kibana/bin/kibana

Restart=on-failure
RestartSec=3

StartLimitBurst=3
StartLimitInterval=60

WorkingDirectory=/usr/share/kibana

StandardOutput=journal
StandardError=inherit

[Install]
WantedBy=multi-user.target


YML:

root@Kibana_server001:/etc/kibana# cat kibana.yml| grep -v "#"
server.port: 5601

server.host: "192.168.13.80"

server.name: "Kibana_server001"

elasticsearch.hosts: ["http://192.168.13.80:9200"]

logging.dest: "/home/user/kibanalog.txt"
elasticsearch.logQueries: false
logging:
  appenders:
    file:
      type: file
      fileName: /var/data/log/kibana/kibana.log
      layout:
        type: pattern
  root:
    appenders: [default,file]



xpack.reporting.roles.enabled: false
xpack.security.session.idleTimeout: "8h"
xpack.security.session.lifespan: "30d"
xpack.encryptedSavedObjects.encryptionKey: DhGFjqLmrjwFvd6AMeF8p2H4dDVK8FXnJeTUb8rBwAMA
xpack.actions.allowedHosts: ["*"]
xpack.actions.enabledActionTypes: ["*"]
logging.verbose: false

xpack.security.enabled: false
xpack.reporting.enabled: false
xpack.spaces.enabled: false


Mar 08 11:45:17 Kibana_server001 kibana[3248031]: {"log.level":"info","@timestamp":"2024-03-08T11:45:17.579Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.2.0","env":{"pid":3248031,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 5.4.0-153-generic","arch":"x64","host":"Kibana_server001","timezone":"UTC+00","runtime":"Node.js v18.18.2"},"config":{"active":{"source":"start","value":true},"breakdownMetrics":{"source":"start","value":false},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false},"centralConfig":{"source":"start","value":false},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["kibana_uuid","bb725f59-470e-4b28-935a-e97f7de98f52"],["git_rev","f5bd489c5ff9c676c4f861c42da6ea99ae350832"]],"sourceValue":{"kibana_uuid":"bb725f59-470e-4b28-935a-e97f7de98f52","git_rev":"f5bd489c5ff9c676c4f861c42da6ea99ae350832"}},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureSpanStackTraces":{"source":"start","sourceValue":false},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.12.2","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.2.0"}
Mar 08 11:45:20 Kibana_server001 kibana[3248031]: Configuring logger failed: ValidationError: [config validation of [logging].dest]: definition for this key is missing
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:94:13)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/src/config_service.js:253:19)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     ... 40 lines matching cause stack trace ...
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:41:25)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:123:47)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:35:31
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/share.js:66:18
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30 {
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:   cause: SchemaTypeError: definition for this key is missing
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ObjectType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:134:14)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.error (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:78:47)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.internals.finalize (/usr/share/kibana/node_modules/joi/lib/validator.js:475:36)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:336:38)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.internals.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:145:28)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.exports.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:27:30)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at internals.Base.validate (/usr/share/kibana/node_modules/joi/lib/base.js:548:26)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/src/types/type.js:89:29)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/src/config_service.js:253:19)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/@kbn/config/src/config_service.js:263:204
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:10:37
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/distinctUntilChanged.js:18:28
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:10:24
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/ReplaySubject.js:54:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:41:25)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:123:47)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:35:31
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/share.js:66:18
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:9:16
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/distinctUntilChanged.js:13:16
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/map.js:9:16
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.<anonymous> (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/lift.js:14:28)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:30:30
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/switchMap.js:17:63
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/ReplaySubject.js:54:24)
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:       at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Observable.js:41:25) {
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     cause: undefined,
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:     path: [ 'dest' ]
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:   }
Mar 08 11:45:20 Kibana_server001 kibana[3248031]: }
Mar 08 11:45:20 Kibana_server001 kibana[3248031]:  FATAL  Error: [config validation of [logging].dest]: definition for this key is missing
Mar 08 11:45:20 Kibana_server001 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 11:45:20 Kibana_server001 systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 08 11:45:23 Kibana_server001 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Mar 08 11:45:23 Kibana_server001 systemd[1]: Stopped Kibana.
Mar 08 11:45:23 Kibana_server001 systemd[1]: kibana.service: Start request repeated too quickly.
Mar 08 11:45:23 Kibana_server001 systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 08 11:45:23 Kibana_server001 systemd[1]: Failed to start Kibana.

I am asking for tips and help.
Thanks in advance.

carly.richmond · March 8, 2024, 1:27pm

Hi @michael8,

Welcome to the community!

The above error message looks similar to that described in this discuss topic. Can you try the solution in that thread and let us know if it resolves your issue?

michael8 · March 11, 2024, 8:52am

Hi,
Unfortunately, the above solution did not help.

Changing ExecStart to ExecStart=/usr/share/kibana/bin/kibana --pid.file=/run/kibana/kibana.pid --deprecation.skip_deprecated_settings[0]=logging.dest

also caused an error:

● kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-03-11 08:49:24 UTC; 1min 33s ago
       Docs: https://www.elastic.co
    Process: 3311270 ExecStart=/usr/share/kibana/bin/kibana --pid.file=/run/kibana/kibana.pid --deprecation.skip_deprecated_settings[0]=logging.dest (code=exited, status=1/FAILURE)
   Main PID: 3311270 (code=exited, status=1/FAILURE)

michael8 · March 11, 2024, 11:37am

I have no more ideas to solve this problem.
I have tried everything from the available discussions. No change helps.

leandrojmp · March 11, 2024, 12:37pm

Did you remove the logging.dest setting from your kibana.yml? This setting was removed on version 8, Kibana will not start if this setting is in your configuration, you need to remove logging.dest from the kibana.yml file.

michael8 · March 11, 2024, 1:26pm

It worked.
I have removed the following entries from kibana.yml

xpack.security.enabled: false
xpack.spaces.enabled: false

Thanks for the help.

system · April 8, 2024, 1:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.