Kibana service fails to start

I have installed Kibana as shown here. Installation was successful and so I began to configure it as shown here by setting the following...

server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]

I did not use the key elasticsearch.url because it has been deprecated. Upon this I have also turned on logging as such...

logging.dest: "/home/syost/kibanalog.txt"
logging.silent: false
logging.quiet: false
logging.verbose: true

After this I have started the service as such...

sudo systemctl start kibana

After I have started the service I keep polling for the status via...

sudo systemctl status kibana

Polling four times approximately 2 seconds apart results in...

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-19 19:59:41 MDT; 4s ago
 Main PID: 5833 (node)
   CGroup: /system.slice/kibana.service
           └─5833 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Apr 19 19:59:41 a-s6bac55wo88z systemd[1]: Started Kibana.
Apr 19 19:59:41 a-s6bac55wo88z systemd[1]: Starting Kibana...

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2020-04-19 19:59:45 MDT; 2s ago
  Process: 5833 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)
 Main PID: 5833 (code=exited, status=1/FAILURE)

Apr 19 19:59:45 a-s6bac55wo88z systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Apr 19 19:59:45 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 19:59:45 a-s6bac55wo88z systemd[1]: kibana.service failed.

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-19 19:59:48 MDT; 3s ago
 Main PID: 5864 (node)
   CGroup: /system.slice/kibana.service
           └─5864 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Apr 19 19:59:48 a-s6bac55wo88z systemd[1]: Started Kibana.
Apr 19 19:59:48 a-s6bac55wo88z systemd[1]: Starting Kibana...

[/etc] > sudo systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2020-04-19 19:59:56 MDT; 2s ago
  Process: 5864 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)
 Main PID: 5864 (code=exited, status=1/FAILURE)

Apr 19 19:59:53 a-s6bac55wo88z systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Apr 19 19:59:53 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 19:59:53 a-s6bac55wo88z systemd[1]: kibana.service failed.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: kibana.service holdoff time over, scheduling restart.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: start request repeated too quickly for kibana.service
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: Failed to start Kibana.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 19:59:56 a-s6bac55wo88z systemd[1]: kibana.service failed.

Logging never appears in /home/syost/kibanalog.txt and so I have nothing to go off of in determining why systemctl failed to start kibana service.

I'm pasting in my entire config file should you see something I don't...

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
logging.dest: "/home/syost/kibanalog.txt"

# Set the value of this setting to true to suppress all logging output.
logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: true

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

It looks like you have used a DEB/RPM installer, so you should have logs in /var/log/kibana?

I installed via sudo yum install kibana. I've went into /var/log/ and there is no kibana directory...

[/var/log] > ll
total 57376
drwxr-xr-x  3 root          root                  17 Apr  1 08:58 amazon
drwx------  2 root          root                  23 Aug 17  2018 audit
-rw-------  1 root          root                4646 Apr 19 17:52 boot.log
-rw-------  1 root          utmp                   0 Apr  1 09:00 btmp
drwxr-xr-x  2 chrony        chrony                 6 Feb 21  2019 chrony
-rw-r--r--  1 root          root              868480 Apr 19 17:53 cloud-init.log
-rw-r--r--  1 root          root              526113 Apr 19 17:53 cloud-init-output.log
-rw-------  1 root          root                5187 Apr 19 18:01 cron
-rw-------  1 root          root               36805 Apr  5 00:01 cron-20200405
-rw-------  1 root          root               52062 Apr 12 00:01 cron-20200412
-rw-------  1 root          root               76912 Apr 19 00:01 cron-20200419
-rw-r--r--  1 root          root               28895 Apr 19 17:52 dmesg
-rw-r--r--  1 root          root               28895 Apr 19 17:37 dmesg.old
drwxr-s---  2 elasticsearch elasticsearch       4096 Apr 19 17:53 elasticsearch
-rw-------  1 root          root                5429 Apr 19 01:51 grubby
-rw-r--r--  1 root          root                 489 Apr  1 09:00 grubby_prune_debug
drwxr-sr-x+ 3 root          systemd-journal       46 Apr  1 08:53 journal
-rw-r--r--  1 root          root              292292 Apr 19 16:47 lastlog
-rw-------  1 root          root                2489 Apr 19 17:52 maillog
-rw-------  1 root          root                 939 Apr  1 09:06 maillog-20200405
-rw-------  1 root          root                   0 Apr  5 00:01 maillog-20200412
-rw-------  1 root          root                   0 Apr 12 00:01 maillog-20200419
-rw-------  1 root          root             2151719 Apr 19 18:15 messages
-rw-------  1 root          root            11981728 Apr  5 00:01 messages-20200405
-rw-------  1 root          root            16889796 Apr 12 00:01 messages-20200412
-rw-------  1 root          root            25539604 Apr 19 00:01 messages-20200419
drwxr-xr-x  4 pcoip         pcoip               4096 Apr 19 18:15 pcoip-agent
drwx------  4 root          root                 183 Apr  1 15:44 samba
-rw-------  1 root          root               51526 Apr 19 18:05 secure
-rw-------  1 root          root               18873 Apr  4 14:38 secure-20200405
-rw-------  1 root          root                9456 Apr  9 14:18 secure-20200412
-rw-------  1 root          root                1580 Apr 13 16:51 secure-20200419
drwxr-xr-x  5 root          root                4096 Apr 19 18:15 skylight
-rw-------  1 root          root                   0 Apr 19 00:01 spooler
-rw-------  1 root          root                   0 Jun 22  2018 spooler-20200405
-rw-------  1 root          root                   0 Apr  5 00:01 spooler-20200412
-rw-------  1 root          root                   0 Apr 12 00:01 spooler-20200419
-rw-------  1 root          root                   0 Jun 22  2018 tallylog
-rw-rw-r--  1 root          utmp               67200 Apr 19 17:54 wtmp
-rw-r--r--  1 root          root               16973 Apr 19 17:53 Xorg.100.log
-rw-r--r--  1 root          root               18149 Apr 19 17:52 Xorg.100.log.old
-rw-------  1 root          root              115086 Apr 19 16:47 yum.log

UPDATE: SOLVED!!!

It appears that when you set a log file destination in the kibana configuration file you cannot just say...

logging.dest: "/home/username/kibanalog.txt"

If you do this while tailing systemctl output via sudo journalctl -fu kibana.service you will see that the service fails to create and open this file. Here is what the error looks like while tailing journalctl output...

Apr 19 20:16:45 a-s6bac55wo88z systemd[1]: Started Kibana.
Apr 19 20:16:45 a-s6bac55wo88z systemd[1]: Starting Kibana...
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: events.js:174
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: throw er; // Unhandled 'error' event
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: ^
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: Error: EACCES: permission denied, open '/home/syost/kibanalog.txt'
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: Emitted 'error' event at:
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at errorOrDestroy (internal/streams/destroy.js:107:12)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at WriteStream.onerror (_stream_readable.js:734:7)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at WriteStream.emit (events.js:198:13)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at lazyFs.open (internal/fs/streams.js:277:12)
Apr 19 20:16:50 a-s6bac55wo88z kibana[6684]: at FSReqWrap.args [as oncomplete] (fs.js:140:20)
Apr 19 20:16:50 a-s6bac55wo88z systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Apr 19 20:16:50 a-s6bac55wo88z systemd[1]: Unit kibana.service entered failed state.
Apr 19 20:16:50 a-s6bac55wo88z systemd[1]: kibana.service failed.
Apr 19 20:16:53 a-s6bac55wo88z systemd[1]: kibana.service holdoff time over, scheduling restart.

The solution I took was to create a file in /usr/share/kibana called kibanalog.txt and then change the ownership (both USER and GROUP) of that file.

[/usr/share/kibana] > sudo touch kibanalog.txt
[/usr/share/kibana] > sudo chown kibana:kibana kibanalog.txt

After doing this I updated my kibana configuration file to reflect this new change as such...

logging.dest: "/usr/share/kibana/kibanalog.txt"

Now when restarting the kibana service via sudo systemctl restart kibana it starts and runs perfectly. Also I verified that kibana is listening on port 5601 to make sure there were no port conflicts via netstat -plntu and it was...

[/usr/share/kibana] > netstat -plntu
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      -                   
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      -                   

I'm not sure why kibana is acting as it's own user. If someone could enlighten me on why Kibana does this that would be helpful. This creates headaches with permissions.

2 Likes

Ahh, you can't mark the original post in a topic as the solution, as it's also the question. Discourse expects a follow up post as a solution, so if it's not too much trouble you could remove it from the first one and add it as another reply, then mark that as a solution :slight_smile:

1 Like

@warkolm
No problem, post has been updated. If you would be so kind could you elaborate on why I have to chown kibana:kibana files to change their ownership to USER: kibana and GROUP: kibana as it's confusing why kibana is a user here.

1 Like

Because Kibana runs as that dedicated user.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.