Kibana service keeps failing after upgrade from 7.17 to 8.7

Elastic Stack Kibana
1

Hi,
I'm trying to upgrade my cluster from 7.17.7 to 8.7 and I'm testing the upgrade on a single node cluster.
I treated all the errors found by the upgrade assistant and upgraded elasticsearch just fine and tried without upgrading kibana and all works fine.

Kibana assistant API results

curl -ks -u user:password https://kibana_host:kibana_port/api/upgrade_assistant/status?pretty
{"readyForUpgrade":true,"details":"All deprecation warnings have been resolved."}

After I upgraded kibana, the kibana service is stoping repeatedly and when I try to run in using

sudo -u kibana /usr/share/kibana/bin/kibana

kibana is working fine and connected to elasticsearch and everything.
The error I'm getting is the following

{"@timestamp":"2023-05-10T09:57:53.680Z","event":{"provider":"eventLog","action":"stopping"},"message":"eventLog stopping","ecs":{"version":"1.8.0"},"kibana":{"server_uuid":"190aeaf3-8ecc-4ea1-bf95-175946e623f8","version":"8.7.1"}}

[2023-05-10T09:57:53.686+00:00][ERROR][plugins.eventLog] Error: Error writing some bulk events
    at ClusterClientAdapter.indexDocuments (/usr/share/kibana/x-pack/plugins/event_log/server/es/cluster_client_adapter.js:86:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at /usr/share/kibana/x-pack/plugins/event_log/server/es/cluster_client_adapter.js:44:216
[
  {
    create: {
      _index: '.kibana-event-log-8.7.1',
      _id: 'bjAYBYgBkRNw1UIoh9aT',
      status: 403,
      error: {
        type: 'cluster_block_exception',
        reason: 'index [.kibana-event-log-8.7.1-000001] blocked by: [FORBIDDEN/8/index write (api)];'
      }
    }
  }
]

I would like to note that I enabled and configured the ssl for all elements before the upgrade so It's not using elastic 8 auto-configured security.

Do you have any recommandations for my situation ?
Also If you have found any challenges with your upgrades from 7.17 to 8 please share them with me.

2

Hello everyone,
Alhamdulillah,
I found the solution for this issue, after installing kibana 8.7.1 on another host and comparing I foundout that the /etc/systemd/system/kibana.service file has been updated in the newer version but installing the upgrade using apt did not handel this. please find below the old and new service definitions.

Kibana 7.17.7 service

[Unit]
Description=Kibana

[Service]
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/

[Install]
WantedBy=multi-user.target

Kibana 8.7.1 service

# /lib/systemd/system/kibana.service
[Unit]
Description=Kibana
Documentation=https://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=kibana
Group=kibana
PrivateTmp=true

Environment=KBN_HOME=/usr/share/kibana
Environment=KBN_PATH_CONF=/etc/kibana

EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana

ExecStart=/usr/share/kibana/bin/kibana

Restart=on-failure
RestartSec=3

StartLimitBurst=3
StartLimitInterval=60

WorkingDirectory=/usr/share/kibana

StandardOutput=journal
StandardError=inherit

[Install]
WantedBy=multi-user.target

After updating the file you will need to run the command

systemctl daemon-reload

for the kibana.yml the only change needed is the new logging configuration as mentioned on the upgrade documentation.

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file

I hope this would be helpfull.
Thank you.

3

Thanks for sharing the solution! :smiley: