Kibana server is not ready yet

[justintaylor9 (Justin Taylor)]

I am getting this error in the browser. Here are my configs:

Kibana.yml:

</>
server.host: 10.241.89.153
Elasticsearch.hosts: "http://10.241.89.144:9200"
server.port: 80
Elasticsearch.ssl.verificationMode: certificate
Elasticsearch.username: kibanaserver
Elasticsearch.password: kibanaserver
Elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
opendistro_security.multitenancy.enabled: true
opendistro_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: false
server.ssl.key: "/etc/kibana/certs/kibana-key.pem"
server.ssl.certificate: "/etc/kibana/certs/kibana.pem"
Elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/root-ca.pem"]
server.defaultRoute: /app/wazuh?security_tenant=global
telemetry.banner: false
</>

(SSL disabled)

Elastic.yml:

</>network.host: 0.0.0.0
http.port: 9200
node.name: node-01
cluster.name: elastic_cluster_test
cluster.initial_master_nodes:
- node-01
discovery.seed_hosts:
- 10.241.89.144
- 10.241.89.154
- 10.241.89.145
node.master: true
node.data: true
node.ingest: true

opendistro_security.disabled: true
opendistro_security.ssl.transport.pemcert_filepath: /etc/Elasticsearch/certs/Elasticsearch.pem
opendistro_security.ssl.transport.pemkey_filepath: /etc/Elasticsearch/certs/Elasticsearch-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: /etc/Elasticsearch/certs/root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false
opendistro_security.ssl.http.enabled: false
opendistro_security.ssl.http.pemcert_filepath: /etc/Elasticsearch/certs/Elasticsearch.pem
opendistro_security.ssl.http.pemkey_filepath: /etc/Elasticsearch/certs/Elasticsearch-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: /etc/Elasticsearch/certs/root-ca.pem
opendistro_security.nodes_dn:

• CN=node-1,OU=UF,O=Wazuh-test,L=Florida,C=US
• CN=node-2,OU=UF,O=Wazuh-test,L=Florida,C=US
• CN=node-3,OU=UF,O=Wazuh-test,L=Florida,C=US
  opendistro_security.authcz.admin_dn:
• CN=admin,OU=Docu,O=Wazuh,L=California,C=US

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
node.max_local_storage_nodes: 3

path.data: /var/lib/Elasticsearch
path.logs: /var/log/Elasticsearch</>

(SSL disabled)

Both Kibana and Elasticsearch are running fine, no errors.

I've reviewed several other similar forum topics but none of the fixes worked for me.

[FALEN (Ali Can Saykal)]

Please enable debug level logs and share errors, warns you found

Logging | Elasticsearch Guide [8.1] | Elastic

Its probably related to Elasticsearch, but you can also find kibana logs here;

Logging settings in Kibana | Kibana Guide [8.1] | Elastic

[justintaylor9 (Justin Taylor)]

Hello:

When I add the logging settings to my kibana.yml the service fails to start:

server.host: 10.241.89.153

Elasticsearch.hosts: http://10.241.89.144:9200

server.port: 80

Elasticsearch.ssl.verificationMode: certificate

Elasticsearch.username: kibanaserver

Elasticsearch.password: kibanaserver

Elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opendistro_security.multitenancy.enabled: true

opendistro_security.readonly_mode.roles: ["kibana_read_only"]

server.ssl.enabled: false

server.ssl.key: "/etc/kibana/certs/kibana-key.pem"

server.ssl.certificate: "/etc/kibana/certs/kibana.pem"

Elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/root-ca.pem"]

server.defaultRoute: /app/wazuh?security_tenant=global

telemetry.banner: false

logging.root.level: debug

logging.appenders.default:

type: file

fileName: /var/logs/kibana.log

layout:

type: json

logging.loggers:

• name: Elasticsearch.query

level: debug

logging.loggers:

• name: http.server.response

level: debug

logging.loggers:

• name: metrics.ops

level: debug

[FALEN (Ali Can Saykal)]

@justintaylor9

server.port: 80
Elasticsearch.ssl.verificationMode: certificate
server.ssl.enabled: false
server.ssl.key: "/etc/kibana/certs/kibana-key.pem"
server.ssl.certificate: "/etc/kibana/certs/kibana.pem"
Elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/root-ca.pem"]

You have both SSL config and non-ssl config enabled. Im not sure that will cause problem but it seems wrong to me

[justintaylor9 (Justin Taylor)]

I’ve disabled SSL completely, it wasn’t working to begin with, so I wanted to deploy this as a POC without it. With SSL disabled and the logging commented out I notice the following…

The service starts and runs for a few minutes … fails then restarts … runs for a few minutes… spits out this message…

Process: 795890 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)

Main PID: 795890 (code=exited, status=1/FAILURE)

Restarts again … seems ok for a few minutes … fails again.

[FALEN (Ali Can Saykal)]

Can you share debug logs of Elasticsearch, just restart service and collect fresh logs and see whats happening right before ERROR happens

[justintaylor9 (Justin Taylor)]

I removed the auto-restart from the kibana.service config to see the error message, it is as follows:

'{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [_opendistro]",

[FALEN (Ali Can Saykal)]

I assume you are missing alerting plugin with kibana, but i cant help further because your Elasticsearch is opendistro and i dont have knowledge on their tools. You may look further details on opendistro forums

[system]

Opendistro is an AWS run product and differs from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )