Why does Kibana stops after 10seconds?

Dear all,

I really didn't want to post a question here, and i am trying for a week to fix my issue, but i can't seem to figure this out.

I am trying to do some tests on Elastic Search Endpoint Security (using elastic as EDR) and every time I am trying to set it up, something goes wrong. I tried on my HP dl380 g5 running ESXI but no luck.

Now I am trying on my host machine, on VMware Workstation 15 pro(15.5.1) running on ubuntu Server 20.0 following almost exactly what this guide is saying.

I think I have Elasticsearch running probably fine, and I have almost completed Kibana installation. The problem is that Kibana service only runs for about 10 seconds and then it stops.

Ubuntu IP: 192.168.1.35

When I visit https://192.168.1.35:9200 as a response I am getting the message:

{
"name" : "elastic",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "HI1740PGQN-qJDsYqB6qcA",
"version" : {
"number" : "7.15.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "83c34f456ae29d60e94d886e455e6a3409bba9ed",
"build_date" : "2021-10-07T21:56:19.031608185Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

From the guide I am following, I believe he has forgotten probably to add some things that we need to uncomment. These are:

" server.port:5601"
and also add
"discovery.seed_hosts: [ ]"

The former was a trial and error from the post here but no luck with this also.

In kibana.yml the only settings I have uncommented are:

server.port: 5601
server.host: "192.168.1.35"
elasticsearch.hosts: ["https://192.168.1.35:9200"]
elasticsearch.username: "kibana_system"
server.ssl.enabled: true
server.ssl.keystore.path: /etc/kibana/http.p12
server.ssl.keystore.password: "123456"

elasticsearch.ssl.certificateAuthorities: /etc/kibana/elasticsearch-ca.pem

xpack.encryptedSavedObjects.encryptionKey: "salkdjfhasldfkjhasdlfkjhasdflkasjdfhslkajfhasldkfjhasdlaksdjfh"




With these settings when I am trying to restart Kibana service I get no error messages in the terminal. If I check in the first 10 seconds the status of the service it will say "active(running)" but about 10s after I get the message:

kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-08-25 01:22:39 UTC; 56s ago
Docs: https://www.elastic.co
Process: 11326 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid (code=exited, status=78)
Main PID: 11326 (code=exited, status=78)

Aug 25 01:22:39 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Aug 25 01:22:39 elastic systemd[1]: Stopped Kibana.
Aug 25 01:22:39 elastic systemd[1]: kibana.service: Start request repeated too quickly.
Aug 25 01:22:39 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 01:22:39 elastic systemd[1]: Failed to start Kibana.

Others errors I get are:

● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-08-25 01:40:58 UTC; 7s ago
Docs: https://www.elastic.co
Process: 12368 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid (code=exited, status=1/FAILURE)
Main PID: 12368 (code=exited, status=1/FAILURE)

Aug 25 01:40:55 elastic kibana[12368]: column: 0
Aug 25 01:40:55 elastic kibana[12368]: }
Aug 25 01:40:55 elastic kibana[12368]: }
Aug 25 01:40:55 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 01:40:55 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 01:40:58 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Aug 25 01:40:58 elastic systemd[1]: Stopped Kibana.
Aug 25 01:40:58 elastic systemd[1]: kibana.service: Start request repeated too quickly.
Aug 25 01:40:58 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 01:40:58 elastic systemd[1]: Failed to start Kibana.

One of the errors was that "FATAL Error: [config validation of [server].ssl.keystore.password]: expected value of type [string] but got [number]"

And after this thread
I tried making some changes on the name "ELASTICSEARCH_HOSTS=["https://X.X.X.X:XXXX"]"
but with no luck.

Now If I run sudo journalctl --unit kibana -f
I get a results:

-- Logs begin at Mon 2022-08-22 19:56:30 UTC. --
Aug 25 01:40:55 elastic kibana[12368]: column: 0
Aug 25 01:40:55 elastic kibana[12368]: }
Aug 25 01:40:55 elastic kibana[12368]: }
Aug 25 01:40:55 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 01:40:55 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 01:40:58 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Aug 25 01:40:58 elastic systemd[1]: Stopped Kibana.
Aug 25 01:40:58 elastic systemd[1]: kibana.service: Start request repeated too quickly.
Aug 25 01:40:58 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 01:40:58 elastic systemd[1]: Failed to start Kibana.

Edit: My elasticsearch.yml is the following:

path.logs: /var/log/elasticsearch
network.host: ["ens33", "local"]

http.port: 9200

discovery.type: single-node

xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

I am still a noob-ie trying to learn so please be kind :stuck_out_tongue:

Does anyone with more experience than me has any idea what I might be doing wrong?
thank you

Welcome to our community! :smiley:

What does /varlog/kibana/kibana.log show?

Hmmmm

server.ssl.keystore.password: "123456"

Seems like have something to do with the that...

Wondering if when you created The cert if you actually used 123456 as the password somehow that get turned into an int and now you're trying to compare an int and a string..

Complete wild guess but interesting nonetheless.

But yes, agree with @warkolm show us the folk about analogs either through var/log or journalctl

hello everyone and thank you for the comments.

It is weird as /kibana.log is empty, and remains empty when I am trying for example to restart the service.

journalctl returns the following. I guess I am missing a path? :stuck_out_tongue:

Aug 25 14:46:14 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:14 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:14 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:14 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:14 elastic systemd[1]: Starting Daily apt upgrade and clean activities...
Aug 25 14:46:18 elastic systemd[1]: apt-daily-upgrade.service: Succeeded.
Aug 25 14:46:18 elastic systemd[1]: Finished Daily apt upgrade and clean activities.
Aug 25 14:46:19 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:19 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:19 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:19 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:24 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:24 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:24 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:24 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:29 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:29 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:29 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:29 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:34 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:34 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:34 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:34 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:39 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:39 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:39 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:39 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:44 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:44 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:44 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:44 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:49 elastic systemd[1]: Starting Refresh fwupd metadata and update motd...
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Activating via systemd: service name='org.freedesktop.fwupd' unit='fwupd.service' requested by ':1.30' (uid=62803 pid=14877 comm="/usr/bin/fwupdmgr refresh " l>
Aug 25 14:46:49 elastic systemd[1]: Starting Firmware update daemon...
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Activating via systemd: service name='org.freedesktop.UPower' unit='upower.service' requested by ':1.31' (uid=0 pid=14889 comm="/usr/libexec/fwupd/fwupd " labe>
Aug 25 14:46:49 elastic systemd[1]: Starting Daemon for power management...
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Successfully activated service 'org.freedesktop.UPower'
Aug 25 14:46:49 elastic systemd[1]: Started Daemon for power management.
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Successfully activated service 'org.freedesktop.fwupd'
Aug 25 14:46:49 elastic systemd[1]: Started Firmware update daemon.
Aug 25 14:46:49 elastic systemd[1]: fwupd-refresh.service: Succeeded.
Aug 25 14:46:49 elastic systemd[1]: Finished Refresh fwupd metadata and update motd.
Aug 25 14:46:52 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:52 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:52 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:52 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:57 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:57 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:57 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:57 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:57 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:57 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:02 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:02 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:02 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:02 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:07 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:07 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:07 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:07 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:12 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:12 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:12 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:12 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:17 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:17 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:17 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:17 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:22 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:22 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:22 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:22 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:27 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:27 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:27 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:27 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:32 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:32 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:32 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:32 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:37 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:37 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:37 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:37 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:42 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:42 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:42 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:42 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:47 elastic multipathd[769]: sda: add missing path
Aug 25 14:47:47 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:47:47 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:47:47 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:47:52 elastic multipathd[769]: sda: add missing path

In order to test this, i left the password blank on the kibana.yml to see if I will be getting wrong authentication message or maybe i left the pass blank. the results are the following

Aug 25 15:28:18 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu

-- Automatic restarting of the unit kibana.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Aug 25 15:28:18 elastic systemd[1]: Stopped Kibana.
-- Subject: A stop job for unit kibana.service has finished
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu

-- A stop job for unit kibana.service has finished.

-- The job identifier is 776 and the job result is done.
Aug 25 15:28:18 elastic systemd[1]: kibana.service: Start request repeated too quickly.
Aug 25 15:28:18 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu

-- The unit kibana.service has entered the 'failed' state with result 'exit-code'.
Aug 25 15:28:18 elastic systemd[1]: Failed to start Kibana.
-- Subject: A start job for unit kibana.service has failed
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu

-- A start job for unit kibana.service has finished with a failure.

-- The job identifier is 776 and the job result is failed.
Aug 25 15:28:21 elastic sudo[1552]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/systemctl status kibana.service
Aug 25 15:28:21 elastic sudo[1552]: pam_unix(sudo:session): session opened for user root by george(uid=0)
Aug 25 15:28:21 elastic sudo[1552]: pam_unix(sudo:session): session closed for user root
Aug 25 15:28:22 elastic multipathd[766]: sda: add missing path
Aug 25 15:28:22 elastic multipathd[766]: sda: failed to get udev uid: Invalid argument
Aug 25 15:28:22 elastic multipathd[766]: sda: failed to get sysfs uid: Invalid argument
Aug 25 15:28:22 elastic multipathd[766]: sda: failed to get sgio uid: No such file or directory
Aug 25 15:28:27 elastic multipathd[766]: sda: add missing path
Aug 25 15:28:27 elastic multipathd[766]: sda: failed to get udev uid: Invalid argument
Aug 25 15:28:27 elastic multipathd[766]: sda: failed to get sysfs uid: Invalid argument
Aug 25 15:28:27 elastic multipathd[766]: sda: failed to get sgio uid: No such file or directory
Aug 25 15:28:32 elastic multipathd[766]: sda: add missing path
Aug 25 15:28:32 elastic multipathd[766]: sda: failed to get udev uid: Invalid argument
Aug 25 15:28:32 elastic multipathd[766]: sda: failed to get sysfs uid: Invalid argument
Aug 25 15:28:32 elastic multipathd[766]: sda: failed to get sgio uid: No such file or directory
Aug 25 15:28:37 elastic multipathd[766]: sda: add missing path
Aug 25 15:28:37 elastic multipathd[766]: sda: failed to get udev uid: Invalid argument
Aug 25 15:28:37 elastic multipathd[766]: sda: failed to get sysfs uid: Invalid argument
Aug 25 15:28:37 elastic multipathd[766]: sda: failed to get sgio uid: No such file or directory

Any ideas?

These errors have nothing to do with Kibana or anything from Elastic stack, it is a error from the multipathd daemon, it is pretty common to happen in virtualized environments and it is pretty easy to fix, this is an example on how to do, but again, it is not related to your issue with Kibana.

You need to check in your system log, /var/log/messages or /var/log/syslog for messages that would hint what is the issue with Kibana, look for kibana in the logs.

if you start kibana with systemctl use this per the docs to get the kibana logs...

journalctl -u kibana.service

I laughed hard, sorry my bad.

So, /var/log/messages/kibana has a file kibana.log that is empty and a file named ' tic\ multipathd[769]:\ sda:\ add\ missing\ path"

that contains logs like the previous

Aug 25 14:46:14 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:14 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:14 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:14 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:14 elastic systemd[1]: Starting Daily apt upgrade and clean activities...
Aug 25 14:46:18 elastic systemd[1]: apt-daily-upgrade.service: Succeeded.
Aug 25 14:46:18 elastic systemd[1]: Finished Daily apt upgrade and clean activities.
Aug 25 14:46:19 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:19 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:19 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:19 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:24 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:24 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:24 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:24 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:29 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:29 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:29 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:29 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:34 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:34 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:34 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:34 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:39 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:39 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:39 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:39 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:44 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:44 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:44 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:44 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory
Aug 25 14:46:49 elastic systemd[1]: Starting Refresh fwupd metadata and update motd...
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Activating via systemd: service name='org.freedesktop.fwupd' unit='fwupd.service' requested by ':1.30' (uid=62803 pid=14877 comm="/usr/bin/fwupdmgr refresh " label="unconfined")
Aug 25 14:46:49 elastic systemd[1]: Starting Firmware update daemon...
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Activating via systemd: service name='org.freedesktop.UPower' unit='upower.service' requested by ':1.31' (uid=0 pid=14889 comm="/usr/libexec/fwupd/fwupd " label="unconfined")
Aug 25 14:46:49 elastic systemd[1]: Starting Daemon for power management...
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Successfully activated service 'org.freedesktop.UPower'
Aug 25 14:46:49 elastic systemd[1]: Started Daemon for power management.
Aug 25 14:46:49 elastic dbus-daemon[883]: [system] Successfully activated service 'org.freedesktop.fwupd'
Aug 25 14:46:49 elastic systemd[1]: Started Firmware update daemon.
Aug 25 14:46:49 elastic systemd[1]: fwupd-refresh.service: Succeeded.
Aug 25 14:46:49 elastic systemd[1]: Finished Refresh fwupd metadata and update motd.
Aug 25 14:46:52 elastic multipathd[769]: sda: add missing path
Aug 25 14:46:52 elastic multipathd[769]: sda: failed to get udev uid: Invalid argument
Aug 25 14:46:52 elastic multipathd[769]: sda: failed to get sysfs uid: Invalid argument
Aug 25 14:46:52 elastic multipathd[769]: sda: failed to get sgio uid: No such file or directory

and things that are on the yml config file:

Aug 25 15:00:20 elastic kibana[15720]: '#server.maxPayload: 1048576\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: "# The Kibana server's name. This is used for display purposes.\n" +
Aug 25 15:00:20 elastic kibana[15720]: '#server.name: "your-hostname"\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# The URLs of the Elasticsearch instances to use for all your queries.\n' +
Aug 25 15:00:20 elastic kibana[15720]: 'elasticsearch.hosts:["https://192.168.1.35:9200"]\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# Kibana uses an index in Elasticsearch to store saved searches, visualizations and\n' +
Aug 25 15:00:20 elastic kibana[15720]: "# dashboards. Kibana creates a new index if the index doesn't already exist.\n" +
Aug 25 15:00:20 elastic kibana[15720]: '#kibana.index: ".kibana"\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# The default application to load.\n' +
Aug 25 15:00:20 elastic kibana[15720]: '#kibana.defaultAppId: "home"\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# If your Elasticsearch is protected with basic authentication, these settings provide\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# the username and password that the Kibana server uses to perform maintenance on the Kibana\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# is proxied through the Kibana server.\n' +
Aug 25 15:00:20 elastic kibana[15720]: 'elasticsearch.username: "kibana_system"\n' +
Aug 25 15:00:20 elastic kibana[15720]: '#elasticsearch.password: "pass"\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# Kibana can also authenticate to Elasticsearch via "service account tokens".\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# If may use this token instead of a username/password.\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# elasticsearch.serviceAccountToken: "my_token"\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# These settings enable SSL for outgoing requests from the Kibana server to the browser.\n' +
Aug 25 15:00:20 elastic kibana[15720]: 'server.ssl.enabled: true\n' +
Aug 25 15:00:20 elastic kibana[15720]: 'server.ssl.keystore.path: /etc/kibana/http.p12\n' +
Aug 25 15:00:20 elastic kibana[15720]: 'server.ssl.keystore.password: 189508\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# Optional settings that provide the paths to the PEM-format SSL certificate and key files.\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# These files are used to verify the identity of Kibana to Elasticsearch and are required when\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.\n' +
Aug 25 15:00:20 elastic kibana[15720]: '#elasticsearch.ssl.certificate: /path/to/your/client.crt\n' +
Aug 25 15:00:20 elastic kibana[15720]: '#elasticsearch.ssl.key: /path/to/your/client.key\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# Optional setting that enables you to specify a path to the PEM file for the certificate\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# authority for your Elasticsearch instance.\n' +
Aug 25 15:00:20 elastic kibana[15720]: 'elasticsearch.ssl.certificateAuthorities: /etc/kibana/elasticsearch-ca.pem\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: '# Saved Object Encryption Key : Pick your own\n' +
Aug 25 15:00:20 elastic kibana[15720]: 'xpack.encryptedSavedObjects.encryptionKey: "salkdjfhasldfkjhasdlfkjhasdflkasjdfhslkajfhasldkfjhasdlaksdjfh"\n' +
Aug 25 15:00:20 elastic kibana[15720]: '\n' +
Aug 25 15:00:20 elastic kibana[15720]: "# To disregard the validity of SSL certifi

On the syslog file, regarding kibana I am finding:

Aug 25 17:15:35 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 17:15:35 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 17:15:38 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Aug 25 17:15:38 elastic systemd[1]: Stopped Kibana.
Aug 25 17:15:38 elastic systemd[1]: kibana.service: Start request repeated too quickly.
Aug 25 17:15:38 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 17:15:38 elastic systemd[1]: Failed to start Kibana.
Aug 25 17:15:35 elastic kibana[1498]: /usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:173
Aug 25 17:15:35 elastic kibana[1498]: throw generateError(state, message);
Aug 25 17:15:35 elastic kibana[1498]: ^
Aug 25 17:15:35 elastic kibana[1498]: YAMLException: can not read a block mapping entry; a multiline key may not be an implicit key at line 37, column 1:
Aug 25 17:15:35 elastic kibana[1498]: # Kibana uses an index in Elasti ...
Aug 25 17:15:35 elastic kibana[1498]: ^
Aug 25 17:15:35 elastic kibana[1498]: at generateError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:167:10)
Aug 25 17:15:35 elastic kibana[1498]: at throwError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:173:9)
Aug 25 17:15:35 elastic kibana[1498]: at readBlockMapping (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1073:9)
Aug 25 17:15:35 elastic kibana[1498]: at composeNode (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1359:12)
Aug 25 17:15:35 elastic kibana[1498]: at readDocument (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1525:3)
Aug 25 17:15:35 elastic kibana[1498]: at loadDocuments (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1588:5)
Aug 25 17:15:35 elastic kibana[1498]: at load (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1614:19)
Aug 25 17:15:35 elastic kibana[1498]: at safeLoad (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1637:10)
Aug 25 17:15:35 elastic kibana[1498]: at readYaml (/usr/share/kibana/node_modules/@kbn/apm-config-loader/target_node/utils/read_config.js:25:47)
Aug 25 17:15:35 elastic kibana[1498]: at getConfigFromFiles (/usr/share/kibana/node_modules/@kbn/apm-config-loader/target_node/utils/read_config.js:57:18) {
Aug 25 17:15:35 elastic kibana[1498]: reason: 'can not read a block mapping entry; a multiline key may not be an implicit key',
Aug 25 17:15:35 elastic kibana[1498]: mark: Mark {
Aug 25 17:15:35 elastic kibana[1498]: name: null,

": can not read a block mapping entry; a multiline key may not be an implicit key at line 37, column 1:"
Can that be the problem?

I realized that a stupid thing i did was renaming the server as elastic, and i guess this has led to some confusion on my part while reading logs..

here is the result. sorry for the many postings

-- Logs begin at Mon 2022-08-22 22:56:30 EEST, end at Thu 2022-08-25 17:38:05 EEST. --
Aug 25 03:38:25 elastic systemd[1]: Started Kibana.
Aug 25 03:38:26 elastic kibana[8703]: FATAL Error: [config validation of [server].ssl.keystone]: definition for this key is missing
Aug 25 03:38:26 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 03:38:26 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:38:29 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1.
Aug 25 03:38:29 elastic systemd[1]: Stopped Kibana.
Aug 25 03:38:29 elastic systemd[1]: Started Kibana.
Aug 25 03:38:30 elastic systemd[1]: Stopping Kibana...
Aug 25 03:38:30 elastic systemd[1]: kibana.service: Succeeded.
Aug 25 03:38:30 elastic systemd[1]: Stopped Kibana.
Aug 25 03:38:46 elastic systemd[1]: Started Kibana.
Aug 25 03:38:47 elastic kibana[8782]: FATAL Error: [config validation of [server].ssl.keystone]: definition for this key is missing
Aug 25 03:38:47 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 03:38:47 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:38:50 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1.
Aug 25 03:38:50 elastic systemd[1]: Stopped Kibana.
Aug 25 03:38:50 elastic systemd[1]: Started Kibana.
Aug 25 03:38:52 elastic kibana[8811]: FATAL Error: [config validation of [server].ssl.keystone]: definition for this key is missing
Aug 25 03:38:52 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 03:38:52 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:38:55 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Aug 25 03:38:55 elastic systemd[1]: Stopped Kibana.
Aug 25 03:38:55 elastic systemd[1]: Started Kibana.
Aug 25 03:38:56 elastic kibana[8840]: FATAL Error: [config validation of [server].ssl.keystone]: definition for this key is missing
Aug 25 03:38:56 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 03:38:56 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:38:59 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Aug 25 03:38:59 elastic systemd[1]: Stopped Kibana.
Aug 25 03:38:59 elastic systemd[1]: kibana.service: Start request repeated too quickly.
Aug 25 03:38:59 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:38:59 elastic systemd[1]: Failed to start Kibana.
Aug 25 03:39:42 elastic systemd[1]: kibana.service: Start request repeated too quickly.
Aug 25 03:39:42 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:39:42 elastic systemd[1]: Failed to start Kibana.
Aug 25 03:39:55 elastic systemd[1]: Started Kibana.
Aug 25 03:39:56 elastic kibana[8910]: FATAL Error: [config validation of [server].ssl.keystone]: definition for this key is missing
Aug 25 03:39:56 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 03:39:56 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:40:00 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1.
Aug 25 03:40:00 elastic systemd[1]: Stopped Kibana.
Aug 25 03:40:00 elastic systemd[1]: Started Kibana.
Aug 25 03:40:01 elastic kibana[8935]: FATAL Error: [config validation of [server].ssl.keystone]: definition for this key is missing
Aug 25 03:40:01 elastic systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 03:40:01 elastic systemd[1]: kibana.service: Failed with result 'exit-code'.
Aug 25 03:40:04 elastic systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Aug 25 03:40:04 elastic systemd[1]: Stopped Kibana.
Aug 25 03:40:04 elastic systemd[1]: Started Kibana.
Aug 25 03:40:05 elastic kibana[8963]: FATAL Error: [config validation of [server].ssl.keystone]: definition for this key is missing

typo

keystone

should be
keystore

r

1 Like

that might be a problem as once on my setup I realized, i made that typo.

But now I cant find where is this mistake.

I checked both .yml files under /etc/kibana and /etc/elasticsearch and in all of them it says keystore.

full of noob questions I know..

Can it be that there was another session open on some files? when i vi kibana.yml it says that there is another session edditing it. I pressed delete, but no luck..

No Clue...

Perhaps start from scratch

Also I would suggest trying 8.x ... all the security is set up for OOTB if you follow the quick start guides.

If you want to use 7.X here is a writeup I did for setup up

1 Like

this was the guide I used this time.

I guess 4th time's the charm :stuck_out_tongue:

If that does not solve anything i will start again with 8.x or I will go for another solution.

Thank to everybody that spent their time trying to help me!
kudos

1 Like

Typos are a killer... be careful.

Or try 8.X you can go end to end secure elasiticsearch and kibana in about 5 mins it does it all for you on initial startup.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.