Kibana service fails to start

I have installed Kibana as shown here. Installation was successful and so I began to configure it as shown here by setting the following...

server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]

I did not use the key elasticsearch.url because it has been deprecated. Upon this I have also turned on logging as such...

logging.dest: "/home/syost/kibanalog.txt"
logging.silent: false
logging.quiet: false
logging.verbose: true

After this I have started the service as such...

sudo systemctl start kibana

After I have started the service I keep polling for the status via...

sudo systemctl status kibana

Polling four times approximately 2 seconds apart results in...

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-19 19:59:41 MDT; 4s ago
 Main PID: 5833 (node)
   CGroup: /system.slice/kibana.service
           └─5833 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Apr 19 19:59:41 a-s6bac55wo88z systemd[1]: Started Kibana.
Apr 19 19:59:41 a-s6bac55wo88z systemd[1]: Starting Kibana...

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2020-04-19 19:59:45 MDT; 2s ago
  Process: 5833 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)
 Main PID: 5833 (code=exited, status=1/FAILURE)

Apr 19 19:59:45 a-s6bac55wo88z systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Apr 19 19:59:45 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 19:59:45 a-s6bac55wo88z systemd[1]: kibana.service failed.

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-19 19:59:48 MDT; 3s ago
 Main PID: 5864 (node)
   CGroup: /system.slice/kibana.service
           └─5864 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Apr 19 19:59:48 a-s6bac55wo88z systemd[1]: Started Kibana.
Apr 19 19:59:48 a-s6bac55wo88z systemd[1]: Starting Kibana...

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2020-04-19 19:59:56 MDT; 2s ago
  Process: 5864 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)
 Main PID: 5864 (code=exited, status=1/FAILURE)

Apr 19 19:59:53 a-s6bac55wo88z systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Apr 19 19:59:53 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 19:59:53 a-s6bac55wo88z systemd[1]: kibana.service failed.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: kibana.service holdoff time over, scheduling restart.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: start request repeated too quickly for kibana.service
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: Failed to start Kibana.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: kibana.service failed.

Logging never appears in /home/syost/kibanalog.txt and so I have nothing to go off of in determining why systemctl failed to start kibana service.

I'm pasting in my entire config file should you see something I don't...

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
logging.dest: "/home/syost/kibanalog.txt"

# Set the value of this setting to true to suppress all logging output.
logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: true

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

It looks like you have used a DEB/RPM installer, so you should have logs in /var/log/kibana?

I installed via sudo yum install kibana. I've went into /var/log/ and there is no kibana directory...

[/var/log] > ll
total 57376
drwxr-xr-x  3 root          root                  17 Apr  1 08:58 amazon
drwx------  2 root          root                  23 Aug 17  2018 audit
-rw-------  1 root          root                4646 Apr 19 17:52 boot.log
-rw-------  1 root          utmp                   0 Apr  1 09:00 btmp
drwxr-xr-x  2 chrony        chrony                 6 Feb 21  2019 chrony
-rw-r--r--  1 root          root              868480 Apr 19 17:53 cloud-init.log
-rw-r--r--  1 root          root              526113 Apr 19 17:53 cloud-init-output.log
-rw-------  1 root          root                5187 Apr 19 18:01 cron
-rw-------  1 root          root               36805 Apr  5 00:01 cron-20200405
-rw-------  1 root          root               52062 Apr 12 00:01 cron-20200412
-rw-------  1 root          root               76912 Apr 19 00:01 cron-20200419
-rw-r--r--  1 root          root               28895 Apr 19 17:52 dmesg
-rw-r--r--  1 root          root               28895 Apr 19 17:37 dmesg.old
drwxr-s---  2 elasticsearch elasticsearch       4096 Apr 19 17:53 elasticsearch
-rw-------  1 root          root                5429 Apr 19 01:51 grubby
-rw-r--r--  1 root          root                 489 Apr  1 09:00 grubby_prune_debug
drwxr-sr-x+ 3 root          systemd-journal       46 Apr  1 08:53 journal
-rw-r--r--  1 root          root              292292 Apr 19 16:47 lastlog
-rw-------  1 root          root                2489 Apr 19 17:52 maillog
-rw-------  1 root          root                 939 Apr  1 09:06 maillog-20200405
-rw-------  1 root          root                   0 Apr  5 00:01 maillog-20200412
-rw-------  1 root          root                   0 Apr 12 00:01 maillog-20200419
-rw-------  1 root          root             2151719 Apr 19 18:15 messages
-rw-------  1 root          root            11981728 Apr  5 00:01 messages-20200405
-rw-------  1 root          root            16889796 Apr 12 00:01 messages-20200412
-rw-------  1 root          root            25539604 Apr 19 00:01 messages-20200419
drwxr-xr-x  4 pcoip         pcoip               4096 Apr 19 18:15 pcoip-agent
drwx------  4 root          root                 183 Apr  1 15:44 samba
-rw-------  1 root          root               51526 Apr 19 18:05 secure
-rw-------  1 root          root               18873 Apr  4 14:38 secure-20200405
-rw-------  1 root          root                9456 Apr  9 14:18 secure-20200412
-rw-------  1 root          root                1580 Apr 13 16:51 secure-20200419
drwxr-xr-x  5 root          root                4096 Apr 19 18:15 skylight
-rw-------  1 root          root                   0 Apr 19 00:01 spooler
-rw-------  1 root          root                   0 Jun 22  2018 spooler-20200405
-rw-------  1 root          root                   0 Apr  5 00:01 spooler-20200412
-rw-------  1 root          root                   0 Apr 12 00:01 spooler-20200419
-rw-------  1 root          root                   0 Jun 22  2018 tallylog
-rw-rw-r--  1 root          utmp               67200 Apr 19 17:54 wtmp
-rw-r--r--  1 root          root               16973 Apr 19 17:53 Xorg.100.log
-rw-r--r--  1 root          root               18149 Apr 19 17:52 Xorg.100.log.old
-rw-------  1 root          root              115086 Apr 19 16:47 yum.log

UPDATE: SOLVED!!!

It appears that when you set a log file destination in the kibana configuration file you cannot just say...

logging.dest: "/home/username/kibanalog.txt"

If you do this while tailing systemctl output via sudo journalctl -fu kibana.service you will see that the service fails to create and open this file. Here is what the error looks like while tailing journalctl output...

Apr 19 20:16:45 a-s6bac55wo88z systemd[1]: Started Kibana.
Apr 19 20:16:45 a-s6bac55wo88z systemd[1]: Starting Kibana...
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: events.js:174
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: throw er; // Unhandled 'error' event
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: ^
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: Error: EACCES: permission denied, open '/home/syost/kibanalog.txt'
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: Emitted 'error' event at:
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at errorOrDestroy (internal/streams/destroy.js:107:12)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at WriteStream.onerror (_stream_readable.js:734:7)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at WriteStream.emit (events.js:198:13)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at lazyFs.open (internal/fs/streams.js:277:12)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at FSReqWrap.args [as oncomplete] (fs.js:140:20)
Apr 19 20:16:50 a-s6bac55wo88z systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Apr 19 20:16:50 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 20:16:50 a-s6bac55wo88z systemd[1]: kibana.service failed.
Apr 19 20:16:53 a-s6bac55wo88z systemd[1]: kibana.service holdoff time over, scheduling restart.

The solution I took was to create a file in /usr/share/kibana called kibanalog.txt and then change the ownership (both USER and GROUP) of that file.

[/usr/share/kibana] > sudo touch kibanalog.txt
[/usr/share/kibana] > sudo chown kibana:kibana kibanalog.txt

After doing this I updated my kibana configuration file to reflect this new change as such...

logging.dest: "/usr/share/kibana/kibanalog.txt"

Now when restarting the kibana service via sudo systemctl restart kibana it starts and runs perfectly. Also I verified that kibana is listening on port 5601 to make sure there were no port conflicts via netstat -plntu and it was...

[/usr/share/kibana] > netstat -plntu
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      -                   
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      -                   

I'm not sure why kibana is acting as it's own user. If someone could enlighten me on why Kibana does this that would be helpful. This creates headaches with permissions.

Ahh, you can't mark the original post in a topic as the solution, as it's also the question. Discourse expects a follow up post as a solution, so if it's not too much trouble you could remove it from the first one and add it as another reply, then mark that as a solution :slight_smile:

1 Like

@warkolm
No problem, post has been updated. If you would be so kind could you elaborate on why I have to chown kibana:kibana files to change their ownership to USER: kibana and GROUP: kibana as it's confusing why kibana is a user here.

1 Like

Because Kibana runs as that dedicated user.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.