Kibana - Error: EACCES: permission denied, open '/etc/kibana/kibana.yml'

Peter_M · December 7, 2021, 12:25pm

Hey all,

Fairly new to setting up Kibana, we are setting up on a test environment at the moment, an azure vm using CentOs.

We installed Elasticseach and Kibana with Ansible playbooks, during some troubleshooting we had whilst enabling X-pack (trial) and some security stuff, we have run into an issue in which the kibana.service will not start, we seem to get

Error: EACCES: permission denied, open '/etc/kibana/kibana.yml' everytime it we restart it or start it.`

Here are the file permissions:

-rw-r--r--. 1 root   kibana 1397 Dec  7 11:57 elastic-ca.pem
-rwxrwx---. 1 root   kibana  130 Dec  2 14:04 kibana.keystore
-rwxr-xr-x. 1 kibana kibana 5092 Dec  7 12:08 kibana.yml
-rw-r--r--. 1 root   kibana  216 Nov  4 13:30 node.options

the systemctl status is

● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2021-12-07 12:09:25 UTC; 11min ago
     Docs: https://www.elastic.co
  Process: 12558 ExecStart=/usr/share/kibana/bin/kibana --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid" (code=exited, status=1/FAILURE)
 Main PID: 12558 (code=exited, status=1/FAILURE)

Dec 07 12:09:22 es-vm1 systemd[1]: Unit kibana.service entered failed state.
Dec 07 12:09:22 es-vm1 systemd[1]: kibana.service failed.
Dec 07 12:09:25 es-vm1 systemd[1]: kibana.service holdoff time over, scheduling restart.
Dec 07 12:09:25 es-vm1 systemd[1]: Stopped Kibana.
Dec 07 12:09:25 es-vm1 systemd[1]: start request repeated too quickly for kibana.service
Dec 07 12:09:25 es-vm1 systemd[1]: Failed to start Kibana.
Dec 07 12:09:25 es-vm1 systemd[1]: Unit kibana.service entered failed state.
Dec 07 12:09:25 es-vm1 systemd[1]: kibana.service failed.

Some of the journalctl -u kibana -r stuff

Dec 07 12:09:25 es-vm1 systemd[1]: kibana.service failed.
Dec 07 12:09:25 es-vm1 systemd[1]: Unit kibana.service entered failed state.
Dec 07 12:09:25 es-vm1 systemd[1]: Failed to start Kibana.
Dec 07 12:09:25 es-vm1 systemd[1]: start request repeated too quickly for kibana.service
Dec 07 12:09:25 es-vm1 systemd[1]: Stopped Kibana.
Dec 07 12:09:25 es-vm1 systemd[1]: kibana.service holdoff time over, scheduling restart.
Dec 07 12:09:22 es-vm1 systemd[1]: kibana.service failed.
Dec 07 12:09:22 es-vm1 systemd[1]: Unit kibana.service entered failed state.
Dec 07 12:09:22 es-vm1 systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Dec 07 12:09:22 es-vm1 kibana[12558]: throw err;
Dec 07 12:09:22 es-vm1 kibana[12558]: internal/fs/utils.js:332
Dec 07 12:09:22 es-vm1 systemd[1]: Started Kibana.
Dec 07 12:09:22 es-vm1 systemd[1]: Stopped Kibana.
Dec 07 12:09:22 es-vm1 systemd[1]: kibana.service holdoff time over, scheduling restart.
Dec 07 12:09:19 es-vm1 systemd[1]: kibana.service failed.
Dec 07 12:09:19 es-vm1 systemd[1]: Unit kibana.service entered failed state.
Dec 07 12:09:19 es-vm1 systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Dec 07 12:09:19 es-vm1 kibana[12549]: }
Dec 07 12:09:19 es-vm1 kibana[12549]: path: '/etc/kibana/kibana.yml'
Dec 07 12:09:19 es-vm1 kibana[12549]: code: 'EACCES',
Dec 07 12:09:19 es-vm1 kibana[12549]: syscall: 'open',
Dec 07 12:09:19 es-vm1 kibana[12549]: errno: -13,
Dec 07 12:09:19 es-vm1 kibana[12549]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10) {
Dec 07 12:09:19 es-vm1 kibana[12549]: at Module._compile (internal/modules/cjs/loader.js:1085:14)
Dec 07 12:09:19 es-vm1 kibana[12549]: at Object.<anonymous> (/usr/share/kibana/src/cli/dist.js:10:17)
Dec 07 12:09:19 es-vm1 kibana[12549]: at module.exports (/usr/share/kibana/src/cli/apm.js:27:3)
Dec 07 12:09:19 es-vm1 kibana[12549]: at initApm (/usr/share/kibana/node_modules/@kbn/apm-config-loader/target_node/init_apm.js:18:64)
Dec 07 12:09:19 es-vm1 kibana[12549]: at loadConfiguration (/usr/share/kibana/node_modules/@kbn/apm-config-loader/target_node/config_loader.js:30:58)
Dec 07 12:09:19 es-vm1 kibana[12549]: at getConfigFromFiles (/usr/share/kibana/node_modules/@kbn/apm-config-loader/target_node/utils/read_config.js:57:18)
Dec 07 12:09:19 es-vm1 kibana[12549]: at readYaml (/usr/share/kibana/node_modules/@kbn/apm-config-loader/target_node/utils/read_config.js:25:69)
Dec 07 12:09:19 es-vm1 kibana[12549]: at readFileSync (fs.js:393:35)
Dec 07 12:09:19 es-vm1 kibana[12549]: at Object.openSync (fs.js:497:3)
Dec 07 12:09:19 es-vm1 kibana[12549]: Error: EACCES: permission denied, open '/etc/kibana/kibana.yml'
Dec 07 12:09:19 es-vm1 kibana[12549]: ^
Dec 07 12:09:19 es-vm1 kibana[12549]: throw err;
Dec 07 12:09:19 es-vm1 kibana[12549]: internal/fs/utils.js:332

The kibana.yml file is all default other than:

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.hosts: http://localhost:9200

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "elastic"
elasticsearch.password: "asdasd2323"

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-ca.pem" ]

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: true

Any ideas? Thanks!

alfred.sigemo · December 7, 2021, 1:22pm

Hi Peter, have anything appeared in the logfile? (/var/log/kibana/kibana.log)

Peter_M · December 7, 2021, 1:27pm

Hey Alfred,

Here is the result of tail -15 kibana.log

Nothing from today, which is weird? as I have been troubleshooting it today also (and yesterday)

{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":19773,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["warning","plugins","actions"],"pid":19773,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["warning","plugins","alerting"],"pid":19773,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["info","plugins","ruleRegistry"],"pid":19773,"message":"Write is disabled; not installing common resources shared between all indices"}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["info","plugins","ruleRegistry"],"pid":19773,"message":"Write is disabled; not installing resources for index .alerts-observability.uptime.alerts"}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["info","plugins","ruleRegistry"],"pid":19773,"message":"Write is disabled; not installing resources for index .alerts-observability.logs.alerts"}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["info","plugins","ruleRegistry"],"pid":19773,"message":"Write is disabled; not installing resources for index .alerts-observability.metrics.alerts"}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["info","plugins","ruleRegistry"],"pid":19773,"message":"Write is disabled; not installing resources for index .alerts-observability.apm.alerts"}
{"type":"log","@timestamp":"2021-12-06T10:37:05+00:00","tags":["info","savedobjects-service"],"pid":19773,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-12-06T10:37:08+00:00","tags":["error","savedobjects-service"],"pid":19773,"message":"Unable to retrieve version information from Elasticsearch nodes. self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2021-12-06T10:37:13+00:00","tags":["error","savedobjects-service"],"pid":19773,"message":"Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-12-06T10:37:42+00:00","tags":["error","savedobjects-service"],"pid":19773,"message":"Unable to retrieve version information from Elasticsearch nodes. self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2021-12-06T10:49:11+00:00","tags":["info","plugins-system","standard"],"pid":19773,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2021-12-06T10:49:11+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":19773,"message":"Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2021-12-06T10:49:41+00:00","tags":["warning","plugins-system","standard"],"pid":19773,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}

alfred.sigemo · December 7, 2021, 1:31pm

Sounds interesting . Are you able to restart kibana and do the same with kibana.log. Is it possible for you to print the Elasticsearch.log as well?

Peter_M · December 7, 2021, 1:38pm

Hey, interesting and frustrating

Here is Elastic log

tail -30 /var/log/Elasticsearch/Elasticsearch.log

[2021-12-03T08:00:53,775][INFO ][o.e.i.g.GeoIpDownloader  ] [localhost] updated geoip database [GeoLite2-ASN.mmdb]
[2021-12-03T08:00:53,813][INFO ][o.e.i.g.GeoIpDownloader  ] [localhost] updating geoip database [GeoLite2-City.mmdb]
[2021-12-03T08:00:54,523][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [localhost] evicted [0] entries from cache after reloading database [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-ASN.mmdb]
[2021-12-03T08:00:54,523][INFO ][o.e.i.g.DatabaseRegistry ] [localhost] successfully reloaded changed geoip database file [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-ASN.mmdb]
[2021-12-03T08:00:54,938][INFO ][o.e.i.g.DatabaseRegistry ] [localhost] successfully reloaded changed geoip database file [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-City.mmdb]
[2021-12-03T08:01:02,275][INFO ][o.e.i.g.DatabaseRegistry ] [localhost] downloading geoip database [GeoLite2-City.mmdb] to [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-City.mmdb.tmp.gz]
[2021-12-03T08:01:02,319][INFO ][o.e.i.g.GeoIpDownloader  ] [localhost] updated geoip database [GeoLite2-City.mmdb]
[2021-12-03T08:01:02,321][INFO ][o.e.i.g.GeoIpDownloader  ] [localhost] updating geoip database [GeoLite2-Country.mmdb]
[2021-12-03T08:01:04,579][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [localhost] evicted [0] entries from cache after reloading database [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-City.mmdb]
[2021-12-03T08:01:04,579][INFO ][o.e.i.g.DatabaseRegistry ] [localhost] successfully reloaded changed geoip database file [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-City.mmdb]
[2021-12-03T08:01:05,608][INFO ][o.e.i.g.DatabaseRegistry ] [localhost] downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-Country.mmdb.tmp.gz]
[2021-12-03T08:01:05,648][INFO ][o.e.i.g.GeoIpDownloader  ] [localhost] updated geoip database [GeoLite2-Country.mmdb]
[2021-12-03T08:01:05,932][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [localhost] evicted [0] entries from cache after reloading database [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-Country.mmdb]
[2021-12-03T08:01:05,933][INFO ][o.e.i.g.DatabaseRegistry ] [localhost] successfully reloaded changed geoip database file [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-Country.mmdb]
[2021-12-03T08:01:17,205][INFO ][o.e.c.m.MetadataCreateIndexService] [localhost] [.tasks] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2021-12-03T08:01:17,210][INFO ][o.e.c.r.a.AllocationService] [localhost] updating number_of_replicas to [0] for indices [.tasks]
[2021-12-03T08:01:17,569][INFO ][o.e.c.r.a.AllocationService] [localhost] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.tasks][0]]]).
[2021-12-03T08:01:17,657][INFO ][o.e.t.LoggingTaskListener] [localhost] 527 finished with response BulkByScrollResponse[took=668.2ms,timed_out=false,sliceId=null,updated=35,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2021-12-03T08:01:17,666][INFO ][o.e.t.LoggingTaskListener] [localhost] 524 finished with response BulkByScrollResponse[took=360.7ms,timed_out=false,sliceId=null,updated=15,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2021-12-03T14:05:56,347][INFO ][o.e.n.Node               ] [localhost] stopping ...
[2021-12-03T14:05:56,362][INFO ][o.e.x.w.WatcherService   ] [localhost] stopping watch service, reason [shutdown initiated]
[2021-12-03T14:05:56,364][INFO ][o.e.x.w.WatcherLifeCycleService] [localhost] watcher has stopped and shutdown
[2021-12-03T14:05:56,365][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [localhost] [controller/3080] [Main.cc@174] ML controller exiting
[2021-12-03T14:05:56,374][INFO ][o.e.x.m.p.NativeController] [localhost] Native controller process has stopped - no new native processes can be started
[2021-12-03T14:05:56,931][INFO ][o.e.n.Node               ] [localhost] stopped
[2021-12-03T14:05:56,931][INFO ][o.e.n.Node               ] [localhost] closing ...
[2021-12-03T14:05:56,949][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [localhost] evicted [0] entries from cache after reloading database [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-Country.mmdb]
[2021-12-03T14:05:56,950][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [localhost] evicted [0] entries from cache after reloading database [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-ASN.mmdb]
[2021-12-03T14:05:56,950][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [localhost] evicted [0] entries from cache after reloading database [/tmp/geoip-databases/WQ0taV6gTLu30BHdo9UUXw/GeoLite2-City.mmdb]
[2021-12-03T14:05:56,952][INFO ][o.e.n.Node               ] [localhost] closed

I have restarted the kibana service, yet the kibana.log stays the same...

leandrojmp · December 7, 2021, 1:40pm

Did the permissions for the folder /etc/kibana changed for some reason?

alfred.sigemo · December 7, 2021, 1:40pm

Thank you, try do the following from /etc/ and /var/lib :

ls -l|grep -E '(kibana)'

Peter_M · December 7, 2021, 1:42pm

Here is the results:

s -l | grep -E '(kibana)'

drwxr-Sr--.  2 root kibana           128 Dec  7 13:11 kibana

alfred.sigemo · December 7, 2021, 1:44pm

Is that in /etc/ ?

Peter_M · December 7, 2021, 1:45pm

Hey Alfred,

Yes indeed, in /etc/

drwxr-Sr--. 2 root kibana 128 Dec 7 13:11 kibana

Thanks,

Peter_M · December 7, 2021, 1:47pm

Yes that is a possibility during our troubleshooting, the results are posted below:

drwxr-Sr--. 2 root kibana 128 Dec 7 13:11 kibana

alfred.sigemo · December 7, 2021, 1:53pm

Okay, try with: chmod 2750 kibana
from /etc/ directory

Peter_M · December 7, 2021, 2:02pm

Hey,

Okay, now I have, in etc:

drwxr-s---. 2 root kibana 128 Dec 7 13:11 kibana

Inside the /etc/kibana:

-rw-r--r--. 1 root   kibana 1397 Dec  7 11:57 elastic-ca.pem
-rwxrwx---. 1 root   kibana  130 Dec  2 14:04 kibana.keystore
-rwxr-xr-x. 1 kibana kibana 5092 Dec  7 12:08 kibana.yml
-rw-r--r--. 1 root   kibana  216 Nov  4 13:30 node.options

Restarted the service and... it's now working! (well it's running, still some issues to go through but I will look at these separately and come back if need be)

Thanks a lot, guys, is it possible to confirm if the issue was the permissions were incorrect on the /etc/kibana folder? Just want to make sure I understand where we went wrong.

Thanks!

alfred.sigemo · December 7, 2021, 2:04pm

No worries! Well i am 98% sure. I tried recreating the issue on my own server (by chown root:root on kibana in /etc/) and got a very similar error message

leandrojmp · December 7, 2021, 2:16pm

It probably was the permission as the S flag is to not allow the group to execute anything from that directory.

Maybe the internal way of how node.js access that file is to execute the file.

Did you change the permissions in your playbook? The defaults, when installed using package managers for example, are 2750.

system · January 4, 2022, 2:16pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
FATAL Error: EACCES: permission denied, open '/etc/kibana/certs/vm1.local.crt' Kibana	4	6467	August 5, 2019
Error: EACCES: permission denied, open '/usr/share/kibana/cert/elastic-certificates.key' Kibana elastic-stack-monitoring , elastic-stack-security , docker	1	40	August 12, 2024
Updated from Kibana 7.17.17 > 7.17.18 and now fails to Start Kibana	4	195	March 28, 2024
Permission denied just after installation Elasticsearch	6	1075	December 22, 2022
FATAL CLI ERROR Error: EACCES: permission denied, open '/etc/kibana/kibana.yml' Kibana	2	967	October 13, 2022

Kibana - Error: EACCES: permission denied, open '/etc/kibana/kibana.yml'

Related Topics