Ssl error on accessing Kibana interface

ishan.abhinit · May 1, 2023, 2:45pm

I have set up Elasticsearch and Kibana on my server loganalysisclass1.cacr.camp. I have also setup Certbot for loganalysisclass1.cacr.camp using this. I can access Kibana on http://loganalysisclass1.cacr.camp:5601 but when I do https://loganalysisclass1.cacr.camp:5601 I get ERR_SSL_PROTOCOL_ERROR.

Looks like I am serving http on Kibana port 5601. How do I correct this?

Kibana.yml

# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: 0.0.0.0

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"

# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://10.0.98.144:9200"]

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# The maximum number of sockets that can be used for communications with elasticsearch.
# Defaults to `Infinity`.
#elasticsearch.maxSockets: 1024

# Specifies whether Kibana should use compression for communications with elasticsearch
# Defaults to `false`.
#elasticsearch.compression: false

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'
#logging.root.level: debug

# Enables you to specify a file where Kibana stores log output.
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
#  layout:
#    type: json

# Logs queries sent to Elasticsearch.
#logging.loggers:
#  - name: elasticsearch.query
#    level: debug

# Logs http responses.
#logging.loggers:
#  - name: http.server.response
#    level: debug

# Logs system usage information.
#logging.loggers:
#  - name: metrics.ops
#    level: debug

# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data

# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000ms.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR".
#i18n.locale: "en"

# =================== Frequently used (Optional)===================

# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.

# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000 objects per batch.
#migrations.batchSize: 1000

# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
#migrations.maxBatchSizeBytes: 100mb

# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
#migrations.retryAttempts: 15

# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000ms
#unifiedSearch.autocomplete.valueSuggestions.timeout: 1000

# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100_000
#unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000

stephenb · May 1, 2023, 2:52pm

Kibana is not HTTPS by default you need to enable it

Perhaps look at

server.ssl.enabled
| Enables SSL/TLS for inbound connections to Kibana. When set to true, a certificate and its corresponding private key must be provided. These can be specified via server.ssl.keystore.path or the combination of server.ssl.certificate and server.ssl.key.

ishan.abhinit · May 1, 2023, 6:28pm

I have made below changes as per this. Now I get ERR_CONNECTION_REFUSED error. I checked my firewall settings- port 80, 443 and 5601 is accessible.

# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /etc/letsencrypt/live/loganalysisclass1.cacr.camp/cert.pem
server.ssl.certificate: /etc/letsencrypt/live/loganalysisclass1.cacr.camp/chain.pem
server.ssl.key: /etc/letsencrypt/live/loganalysisclass1.cacr.camp/privkey.pem

stephenb · May 1, 2023, 6:42pm

That is not correct....

The second line overwrites the first line.

When I do this with letsencrypt I use the fullchain.pem for the cert ..

You might try concatenating your two into a single file and using that

I'm not the biggest cert expert

Did you try just the first line??

ishan.abhinit · May 1, 2023, 6:51pm

Yes, I did try just the first line, it did not work.
Now, I also tried the fullchain.pem. This did not work either.

stephenb · May 1, 2023, 6:53pm

Does not work means what?

What is the result of

curl -v https://loganalysisclass1.cacr.camp:5601

I Use the fullchain.pem & privkey.pem from letsencrypt successfully on a regular basis.

ishan.abhinit · May 1, 2023, 7:01pm

It means I get the same connection refused error.

Here
curl -v https://loganalysisclass1.cacr.camp:5601

*   Trying 149.165.175.20:5601...
* connect to 149.165.175.20 port 5601 failed: Connection refused
* Failed to connect to loganalysisclass1.cacr.camp port 5601: Connection refused
* Closing connection 0
curl: (7) Failed to connect to loganalysisclass1.cacr.camp port 5601: Connection refused

stephenb · May 1, 2023, 7:03pm

So that is a connectivity issue before the SSL even is considered.

Most Likely Firewall...

Or Kibana is not even running... did you make sure kibana is running?

How are you starting it?

Did you check the logs?

ishan.abhinit · May 1, 2023, 7:30pm

Kibana is set up to start when the server boots up. I restart kibana everytime I make changes to kibana.yml file.
NOTE: I am only working with a log file that has logs from Oct 2019.

Last few lines from /var/log/kibana/kibana.log

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:12.687+00:00","message":"Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:12.717+00:00","message":"config sourced from: production cluster","log":{"level":"INFO","logger":"plugins.monitoring.monitoring"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.286+00:00","message":"http server running at http://0.0.0.0:5601","log":{"level":"INFO","logger":"http.server.Kibana"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.289+00:00","message":"Task Fleet-Usage-Logger-Task scheduled with interval 15m","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.312+00:00","message":"Starting monitoring stats collection","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.388+00:00","message":"Installed common resources shared between all indices","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.388+00:00","message":"Installing resources for index .alerts-observability.uptime.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .preview.alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-observability.logs.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-observability.metrics.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-observability.apm.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.395+00:00","message":"Kibana is now degraded","log":{"level":"INFO","logger":"status"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.406+00:00","message":"Installed resources for index .alerts-observability.apm.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.406+00:00","message":"Installed resources for index .alerts-observability.metrics.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.407+00:00","message":"Installed resources for index .alerts-observability.logs.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.409+00:00","message":"Installed resources for index .alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.410+00:00","message":"Installed resources for index .alerts-observability.uptime.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.418+00:00","message":"Installed resources for index .preview.alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.903+00:00","message":"Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:15.951+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"fd4213663611f1ecaf18d7563e7e903e"},"transaction":{"id":"db13fa27edf5cb2c"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:15.979+00:00","message":"Kibana is now available (was degraded)","log":{"level":"INFO","logger":"status"},"process":{"pid":617342},"trace":{"id":"05097987330621b45e867bc9c4de021c"},"transaction":{"id":"5df4bb1f9efc239b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:13:00.892+00:00","message":"Running Fleet Usage telemetry send task","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"fd4213663611f1ecaf18d7563e7e903e"},"transaction":{"id":"40c7f0ff2c0e44cf"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:25:15.921+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":

stephenb · May 1, 2023, 7:40pm

If this is from the last run

..."message":"http server running at http://0.0.0.0:5601","log":{"level":"INFO","logger":"http.server.Kibana"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}

Which indicates that kibana is still running on HTTP not HTTPS

Which mean this is not properly set.

server.ssl.enabled: true

The logs should look like

...."message":"http server running at https://0.0.0.0:5601",...

BTW this is my exact section with Kibana on HTTPS with let's encrypt full certs

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/fullchain.pem
server.ssl.key: /etc/kibana/privkey.pem

There is still something wrong with the kibana.yml...

Please post your full kibana.yml with the commands you are using to start stop.

There is something simple wrong at this point.

Do you have more than 1 copy of the kibana.yml is the same directory?

ishan.abhinit · May 2, 2023, 1:36pm

I am using Rocky Linux 9 and installed everything as root - Elasticsearch, kibana, filebeat and certbot.

Commands I am using start/restart kibana:-

service kibana start/restart

Kibana used to start automatically on server boot up until yesterday. Since yesterday that is not happening anymore. Now, when I start it manually, it stops after few seconds.

service kibana status

Redirecting to /bin/systemctl status kibana.service
× kibana.service - Kibana
     Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Tue 2023-05-02 13:06:26 UTC; 31s ago
   Duration: 11.027s
       Docs: https://www.elastic.co
    Process: 645735 ExecStart=/usr/share/kibana/bin/kibana (code=exited, status=1/FAILURE)
   Main PID: 645735 (code=exited, status=1/FAILURE)
        CPU: 12.213s

May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: Stopped Kibana.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Consumed 12.213s CPU time.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Start request repeated too quickly.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Failed with result 'exit-code'.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: Failed to start Kibana.

journalctl -fu kibana.service

May 02 13:06:23 loganalysisclass-2023.novalocal kibana[645735]:  FATAL  Error: EACCES: permission denied, open '/etc/letsencrypt/live/loganalysisclass1.cacr.camp/privkey.pem'
May 02 13:06:23 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
May 02 13:06:23 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Failed with result 'exit-code'.
May 02 13:06:23 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Consumed 12.213s CPU time.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: Stopped Kibana.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Consumed 12.213s CPU time.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Start request repeated too quickly.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: kibana.service: Failed with result 'exit-code'.
May 02 13:06:26 loganalysisclass-2023.novalocal systemd[1]: Failed to start Kibana.

Kibana.yml

# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html

# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: 0.0.0.0

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"

# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /etc/letsencrypt/live/loganalysisclass1.cacr.camp/fullchain.pem
server.ssl.key: /etc/letsencrypt/live/loganalysisclass1.cacr.camp/privkey.pem

# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://10.0.98.144:9200"]

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# The maximum number of sockets that can be used for communications with elasticsearch.
# Defaults to `Infinity`.
#elasticsearch.maxSockets: 1024

# Specifies whether Kibana should use compression for communications with elasticsearch
# Defaults to `false`.
#elasticsearch.compression: false

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'
#logging.root.level: debug

# Enables you to specify a file where Kibana stores log output.
logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
#  layout:
#    type: json

 Logs queries sent to Elasticsearch.
#logging.loggers:
#  - name: elasticsearch.query
#    level: debug

# Logs http responses.
#logging.loggers:
#  - name: http.server.response
#    level: debug

# Logs system usage information.
#logging.loggers:
#  - name: metrics.ops
#    level: debug

# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data

# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000ms.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR".
#i18n.locale: "en"

# =================== Frequently used (Optional)===================

# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.

# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000 objects per batch.
#migrations.batchSize: 1000

# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
#migrations.maxBatchSizeBytes: 100mb

# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
#migrations.retryAttempts: 15

# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000ms
#unifiedSearch.autocomplete.valueSuggestions.timeout: 1000

# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100_000
#unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000

Latest logs from /var/log/kibana/kibana.log

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:11.595+00:00","message":"[.kibana] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 0ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:11.596+00:00","message":"[.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 1ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:11.596+00:00","message":"[.kibana] Migration completed after 48ms","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:11.600+00:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:11.601+00:00","message":"Starting [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:12.687+00:00","message":"Task Fleet-Usage-Sender-1.1.0 scheduled with interval 1h","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:12.717+00:00","message":"config sourced from: production cluster","log":{"level":"INFO","logger":"plugins.monitoring.monitoring"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.286+00:00","message":"http server running at http://0.0.0.0:5601","log":{"level":"INFO","logger":"http.server.Kibana"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.289+00:00","message":"Task Fleet-Usage-Logger-Task scheduled with interval 15m","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.312+00:00","message":"Starting monitoring stats collection","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.388+00:00","message":"Installed common resources shared between all indices","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.388+00:00","message":"Installing resources for index .alerts-observability.uptime.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .preview.alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-observability.logs.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-observability.metrics.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.389+00:00","message":"Installing resources for index .alerts-observability.apm.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.395+00:00","message":"Kibana is now degraded","log":{"level":"INFO","logger":"status"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.406+00:00","message":"Installed resources for index .alerts-observability.apm.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.406+00:00","message":"Installed resources for index .alerts-observability.metrics.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.407+00:00","message":"Installed resources for index .alerts-observability.logs.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.409+00:00","message":"Installed resources for index .alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.410+00:00","message":"Installed resources for index .alerts-observability.uptime.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.418+00:00","message":"Installed resources for index .preview.alerts-security.alerts","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:14.903+00:00","message":"Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":617342},"trace":{"id":"0df7330e1e7402a63be6dd702aa86ed4"},"transaction":{"id":"8ef0e72b36256f38"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:15.951+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"fd4213663611f1ecaf18d7563e7e903e"},"transaction":{"id":"db13fa27edf5cb2c"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:10:15.979+00:00","message":"Kibana is now available (was degraded)","log":{"level":"INFO","logger":"status"},"process":{"pid":617342},"trace":{"id":"05097987330621b45e867bc9c4de021c"},"transaction":{"id":"5df4bb1f9efc239b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:13:00.892+00:00","message":"Running Fleet Usage telemetry send task","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"fd4213663611f1ecaf18d7563e7e903e"},"transaction":{"id":"40c7f0ff2c0e44cf"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:25:15.921+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"fd4213663611f1ecaf18d7563e7e903e"},"transaction":{"id":"4d4a43e170a97dac"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:35:31.923+00:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":617342}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:35:31.925+00:00","message":"Monitoring stats collection is stopped","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":617342}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:35:31.938+00:00","message":"Beginning fleet setup","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":617342},"trace":{"id":"b2e67ed99354baee883020c853c9166f"},"transaction":{"id":"d21aba25c17066eb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:35:34.792+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618095},"trace":{"id":"39e65c07bab8668aab31c8dac172430d"},"transaction":{"id":"b43526cd590f7d32"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:35:49.201+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618114},"trace":{"id":"fe91de320fc9c570b779ab54c439e6b8"},"transaction":{"id":"ef4aafe87f8e75cb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:36:03.325+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618184},"trace":{"id":"6cdcd4fe65e079ab678d19ab2298268d"},"transaction":{"id":"4de943be59b40c6b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:37:22.361+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618285},"trace":{"id":"f4f975ff7bdcb6038903ce9342fbf4c9"},"transaction":{"id":"cdb1b9062f53e72a"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:37:36.708+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618308},"trace":{"id":"f2db60942d1fa2c2247e659ac6ffcb26"},"transaction":{"id":"cb5cda07eab4d07c"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:37:50.958+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618326},"trace":{"id":"077909783d5d9179a8d91ffd73b95240"},"transaction":{"id":"24d7a03f2b183f60"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:38:48.781+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618390},"trace":{"id":"8472d55acfaefe9dffa6e4bce83b51a5"},"transaction":{"id":"bc1fe64aade0a303"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:39:02.960+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618408},"trace":{"id":"4d04b375a6f742f533cbdcb1f787812e"},"transaction":{"id":"55227dcce3235276"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:39:17.206+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618447},"trace":{"id":"3f591171dd832985d84493b6e33654f7"},"transaction":{"id":"15659888e9ebd44d"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:54:32.504+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618866},"trace":{"id":"aa6cf7b9ed0c4b8746465d2b946cfdd0"},"transaction":{"id":"090ec7cae2ce9a86"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:54:46.980+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618891},"trace":{"id":"5c495fdd15634c1a9ff79784fd277661"},"transaction":{"id":"74c2ec0ebe3a0dea"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-01T19:55:01.202+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":618909},"trace":{"id":"0905c071fe533726c77fa5c677d73859"},"transaction":{"id":"26a14e2d29569802"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-02T13:04:13.460+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":645563},"trace":{"id":"76692946d999e51e3d5cd2f1eb72eac8"},"transaction":{"id":"e8dbcba15bb37e7a"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-02T13:04:27.984+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":645589},"trace":{"id":"18adb5b65983c3768801b9bbb0dba589"},"transaction":{"id":"95263b686861d8f4"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-02T13:04:41.923+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":645607},"trace":{"id":"96a1600accab2596675750926920e68d"},"transaction":{"id":"9746d5635638c8bb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-02T13:05:45.545+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":645673},"trace":{"id":"a707b7c9b8ab2086c4e41118ccb915d4"},"transaction":{"id":"67476d62f58576f3"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-02T13:05:59.691+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":645696},"trace":{"id":"22181673cd3fbfee206c723e0003465a"},"transaction":{"id":"eb4887086cf81f4b"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-05-02T13:06:13.941+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":645735},"trace":{"id":"5111e53a25f3162ed250a0aeb7b59aea"},"transaction":{"id":"07baeaa46f4bdb8a"}}

Certificate File permissions

lrwxrwxrwx. 1 root root  51 Apr 27 15:14 cert.pem -> ../../archive/loganalysisclass1.cacr.camp/cert1.pem
lrwxrwxrwx. 1 root root  52 Apr 27 15:14 chain.pem -> ../../archive/loganalysisclass1.cacr.camp/chain1.pem
lrwxrwxrwx. 1 root root  56 Apr 27 15:14 fullchain.pem -> ../../archive/loganalysisclass1.cacr.camp/fullchain1.pem
lrwxrwxrwx. 1 root root  54 Apr 27 15:14 privkey.pem -> ../../archive/loganalysisclass1.cacr.camp/privkey1.pem

stephenb · May 2, 2023, 1:39pm

As you can see in Your logs still say HTTP not https.

Try to open Kibana in the browser on HTTP

http://149.165.175.20:5601

Then stop Kibana and run

sudo /bin/systemctl daemon-reload

Then start.

There is something fundamental like you have two installations or something.

Set the loggin level to debug and let's see if we can see something more.

What is this in your kibana.yml It does not appear to be commented out.

 Logs queries sent to Elasticsearch.
#logging

There is something fundamental like you have two installations or something.

ishan.abhinit · May 2, 2023, 4:08pm

I don't think I have two installations. how do i check it anyway?

Adding illegal text at the top of kibana.yml file does fail to start it on the browser. Also, how do I set logging to debug mode?

Could the problem be due to some file permission on certificates like this?

stephenb · May 2, 2023, 4:43pm

Yes it could be permissions but then kibana generally fails to start because it can not open the file

Debug see here

logging.root.level: debug

When you tried to open the running kibana on HTTP what happened? You did not answer that? The logs show it is running on HTTP.

Did you run the daemon reload...

You did not answer about the extra line / uncommented line in the kibana.yml

you are only answering some of my questions/requests.

Something simple is going on... details matter.

Uninstall / reinstall etc...

ishan.abhinit · May 2, 2023, 5:57pm

I already did all that.

I am able to run Kibana in browser in HTTP but only after I uncomment these 3 lines otherwise it does it does not work.

server.ssl.enabled: true
server.ssl.certificate: /etc/letsencrypt/live/loganalysisclass1.cacr.camp/fullchain.pem
server.ssl.key: /etc/letsencrypt/live/loganalysisclass1.cacr.camp/privkey.pem

So I uncommented the above lines and then opened Kibana in browser on HTTP. It works normally. Then stopped kibana and ran

sudo /bin/systemctl daemon-reload

Again started kibana in the browser, it works just fine.

I had also commented out that line in Kibana.yml file as suggested.

stephenb · May 2, 2023, 6:16pm

Ok thanks, for the clarity, I could only guess otherwise

Try putting physical copies of the certs in /etc/kibana make sure the permissions are correct and try the...

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/fullchain.pem
server.ssl.key: /etc/kibana/privkey.pem

Enable debug and look close at the logs

Here is my directory

$ ls -l /etc/kibana/
total 40
-rw-r--r-- 1 root kibana 5636 Jan 11 02:55 fullchain.pem
-rw-rw---- 1 root kibana  130 Jun 27  2022 kibana.keystore
-rw-rw---- 1 root kibana 8457 Mar 17 03:55 kibana.yml
-rw-rw---- 1 root kibana 7634 Jul 23  2022 kibana.yml.dpkg-dist
-rw-r--r-- 1 root kibana  305 Jun  8  2022 node.options
-rw-r--r-- 1 root kibana 1708 Jan 11 02:55 privkey.pem

system · May 2, 2023, 6:21pm

Automated Post Deleted by @stephenb : Wrong Information

ishan.abhinit · May 2, 2023, 7:01pm

Oh that worked. Just copied the certs to /etc/kibana and it worked.

Thank you so so much for your patience and help, Stephen. Really appreciate it.

system · May 30, 2023, 7:01pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.