Kibana Log Error on Start

Elastic Stack Kibana
1

Hello! I recently upgraded my elasticsearch and kibana from 8.3 to 8.11. elasticsearch is running fine but I can not get kibana to start after the upgrade. I installed Kibana via manual Deb and am using the same .yml file from my 8.3 version. I started Kibana via systemctl.

(I took this over from someone who previously setup the elk stack so I tried to keep everything the same).

I get the following error log from journalctl.

Nov 22 12:05:16 sr0005eldev01 kibana[3907267]: error: option '-l, --log-file <path>' argument missing
Nov 22 12:05:16 sr0005eldev01 kibana[3907267]: {"log.level":"info","@timestamp":"2023-11-22T18:05:16.578Z","log":{"logger":"elastic-apm-node"},"agentVersion":"4.0.0","env":{"pid":3907267,">
Nov 22 12:05:15 sr0005eldev01 systemd[1]: Started Kibana.
Nov 22 12:05:15 sr0005eldev01 systemd[1]: kibana.service: Consumed 1.060s CPU time.
Nov 22 12:05:15 sr0005eldev01 systemd[1]: Stopped Kibana.
Nov 22 12:05:15 sr0005eldev01 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Nov 22 12:05:05 sr0005eldev01 systemd[1]: kibana.service: Consumed 1.060s CPU time.
Nov 22 12:05:05 sr0005eldev01 systemd[1]: kibana.service: Failed with result 'exit-code'.
Nov 22 12:05:05 sr0005eldev01 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Nov 22 12:05:05 sr0005eldev01 kibana[3907250]: error: option '-l, --log-file <path>' argument missing
Nov 22 12:05:05 sr0005eldev01 kibana[3907250]: {"log.level":"info","@timestamp":"2023-11-22T18:05:05.611Z","log":{"logger":"elastic-apm-node"},"agentVersion":"4.0.0","env":{"pid":3907250,">
Nov 22 12:05:04 sr0005eldev01 systemd[1]: Started Kibana.
Nov 22 12:05:04 sr0005eldev01 systemd[1]: kibana.service: Consumed 1.056s CPU time.
Nov 22 12:05:04 sr0005eldev01 systemd[1]: Stopped Kibana.
Nov 22 12:05:04 sr0005eldev01 systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1.
Nov 22 12:04:54 sr0005eldev01 systemd[1]: kibana.service: Consumed 1.056s CPU time.
Nov 22 12:04:54 sr0005eldev01 systemd[1]: kibana.service: Failed with result 'exit-code'.
Nov 22 12:04:54 sr0005eldev01 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Nov 22 12:04:54 sr0005eldev01 kibana[3907232]: error: option '-l, --log-file <path>' argument missing
Nov 22 12:04:54 sr0005eldev01 kibana[3907232]: {"log.level":"info","@timestamp":"2023-11-22T18:04:54.606Z","log":{"logger":"elastic-apm-node"},"agentVersion":"4.0.0","env":{"pid":3907232,">
Nov 22 12:04:53 sr0005eldev01 systemd[1]: Started Kibana.

My .yml file has it logging set as follows:

logging:
 appenders:
  file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      -default
      -file

I do not have any apm settings in the .yml. I do have a apm.js and node.options file but they do not appear to have anything in them related to this.

Any guidance would be appreciated. I've beat my head against this for a few hours now.

2

Hi @sgrubb — do you see the same behavior if you start Kibana directly instead of using systemctl?

3

This is what I get when I run it directly:

{"log.level":"info","@timestamp":"2023-11-22T19:03:22.003Z","log":{"logger":"elastic-apm-node"},"agentVersion":"4.0.0","env":{"pid":3907722,"proctitle":"./../node/bin/node","os":"linux 5.15.0-41-generic","arch":"x64","host":"sr0005eldev01","timezone":"UTC-0600","runtime":"Node.js v18.18.2"},"config":{"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.11.1","commonName":"service_version"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"active":{"source":"start","value":true},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["git_rev","09feaf416f986b239b8e8ad95ecdda0f9d56ebec"]],"sourceValue":{"git_rev":"09feaf416f986b239b8e8ad95ecdda0f9d56ebec"}},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"breakdownMetrics":{"source":"start","value":false},"captureSpanStackTraces":{"source":"start","sourceValue":false},"centralConfig":{"source":"start","value":false},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"propagateTracestate":{"source":"start","value":true},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false}},"activationMethod":"require","ecs":{"version":"1.6.0"},"message":"Elastic APM Node.js Agent v4.0.0"}
FATAL CLI ERROR Error: EACCES: permission denied, open '/etc/kibana/kibana.yml'
    at Object.openSync (node:fs:603:3)
    at readFileSync (node:fs:471:35)
    at readYaml (/usr/share/kibana/node_modules/@kbn/config/src/raw/read_config.js:20:69)
    at getConfigFromFiles (/usr/share/kibana/node_modules/@kbn/config/src/raw/read_config.js:56:18)
    at getServerlessModeFromCfg (/usr/share/kibana/src/cli/serve/compile_config_stack.js:70:49)
    at compileConfigStack (/usr/share/kibana/src/cli/serve/compile_config_stack.js:53:64)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:174:66)
    at Command.<anonymous> (/usr/share/kibana/src/cli/command.js:93:20)
    at Command.listener (/usr/share/kibana/node_modules/commander/index.js:370:29)
    at Command.emit (node:events:517:28)
    at Command.parseArgs (/usr/share/kibana/node_modules/commander/index.js:892:12)
    at Command.parse (/usr/share/kibana/node_modules/commander/index.js:642:21)
    at Object.<anonymous> (/usr/share/kibana/src/cli/cli.js:44:9)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Function.Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:188:39)
    at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:188:39)
    at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:188:39)
    at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:188:39)
    at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:188:39)
    at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:188:39)
    at require (node:internal/modules/cjs/helpers:119:18)
    at Object.<anonymous> (/usr/share/kibana/src/cli/dist.js:15:1)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Function.Module._load (node:internal/modules/cjs/loader:960:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12)
    at node:internal/main/run_main_module:23:47

EDIT: Realized i fogot to specify the config file. This is what I get now. I confirmed that Kibana is the owner of the .yml file.

4

Ok I fixed the issue with the .yml but now I am getting this error:

 FATAL  Error: [config validation of [logging].appenders.root]: Cannot destructure property 'Symbol(Symbol.iterator)' of 'undefined' as it is undefined.
5

I fixed the issue. The problem was that the .yml logging settings needed updating and the permissions across several directories needed to be changed. This is likely due to the stack being setup by one user and then essentially reinstalled by me, thus changing some of the permissions.

Closing with solution.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.