i am getting [Kibana server is not ready yet] after reboot server
Started Kibana.
: [2022-06-14T11:08:40.458+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
: [2022-06-14T11:08:40.498+00:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
: [2022-06-14T11:08:40.637+00:00][INFO ][http.server.Preboot] http server running at https://0.0.0.0:5601
: [2022-06-14T11:08:40.711+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
: [2022-06-14T11:08:40.805+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in >
: [2022-06-14T11:08:41.167+00:00][INFO ][plugins-system.standard] Setting up [117] plugins: [translations,monitoringCollection,licens>
: [2022-06-14T11:08:41.199+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 5f28f0e1-ff37-4dd1-aa34-0>
: [2022-06-14T11:08:41.459+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To preven>
: [2022-06-14T11:08:41.502+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To preven>
: [2022-06-14T11:08:41.537+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prev>
: [2022-06-14T11:08:41.540+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting >
: [2022-06-14T11:08:41.559+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely l>
: [2022-06-14T11:08:41.592+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing enc>
: [2022-06-14T11:08:41.631+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing en>
: [2022-06-14T11:08:41.661+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
: [2022-06-14T11:08:42.827+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, >
: [2022-06-14T11:08:42.940+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. conn>
: [2022-06-14T11:08:44.030+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screen>
Stopping Kibana...
: [2022-06-14T11:10:56.217+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
: [2022-06-14T11:10:56.222+00:00][INFO ][plugins-system.standard] Stopping all plugins.
: [2022-06-14T11:10:56.224+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection is stopped
: [2022-06-14T11:11:26.244+00:00][WARN ][plugins-system.standard] "eventLog" plugin didn't stop in 30sec., move on to the next.
: [2022-06-14T11:11:26.269+00:00][FATAL][root] no elements in sequence
: FATAL EmptyError: no elements in sequence
kibana.service: Main process exited, code=exited, status=1/FAILURE
kibana.service: Failed with result 'exit-code'.
Jun 14 11:12:09 wazuh-server kibana[6828]: [2022-06-14T11:12:09.567+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
Jun 14 11:12:09 wazuh-server kibana[6828]: [2022-06-14T11:12:09.608+00:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
Jun 14 11:12:09 wazuh-server kibana[6828]: [2022-06-14T11:12:09.672+00:00][FATAL][root] Error: Unknown configuration key(s): "port". Check for spelling errors and ensure>
Jun 14 11:12:09 wazuh-server kibana[6828]: at ensureValidConfiguration (/usr/share/kibana/src/core/server/config/ensure_valid_configuration.js:35:11)
Jun 14 11:12:09 wazuh-server kibana[6828]: at Server.preboot (/usr/share/kibana/src/core/server/server.js:164:5)
Jun 14 11:12:09 wazuh-server kibana[6828]: at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:14)
Jun 14 11:12:09 wazuh-server kibana[6828]: at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)
Jun 14 11:12:09 wazuh-server kibana[6828]: at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)
Jun 14 11:12:09 wazuh-server kibana[6828]: FATAL Error: Unknown configuration key(s): "port". Check for spelling errors and ensure that expected plugins are installed.
Jun 14 11:12:09 wazuh-server systemd[1]: kibana.service: Main process exited, code=exited, status=64/USAGE
Jun 14 11:12:09 wazuh-server systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 14 11:12:12 wazuh-server systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Jun 14 11:12:12 wazuh-server systemd[1]: Stopped Kibana.
Jun 14 11:12:12 wazuh-server systemd[1]: Started Kibana.
this is the log i get when i run systemctl -u kibana.service
Jun 14 11:12:09 wazuh-server kibana[6828]: [2022-06-14T11:12:09.567+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
Jun 14 11:12:09 wazuh-server kibana[6828]: [2022-06-14T11:12:09.608+00:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
Jun 14 11:12:09 wazuh-server kibana[6828]: [2022-06-14T11:12:09.672+00:00][FATAL][root] Error: Unknown configuration key(s): "port". Check for spelling errors and ensure that expected plugins are installed.
Jun 14 11:12:09 wazuh-server kibana[6828]: at ensureValidConfiguration (/usr/share/kibana/src/core/server/config/ensure_valid_configuration.js:35:11)
Jun 14 11:12:09 wazuh-server kibana[6828]: at Server.preboot (/usr/share/kibana/src/core/server/server.js:164:5)
Jun 14 11:12:09 wazuh-server kibana[6828]: at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:14)
Jun 14 11:12:09 wazuh-server kibana[6828]: at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)
Jun 14 11:12:09 wazuh-server kibana[6828]: at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)
Jun 14 11:12:09 wazuh-server kibana[6828]: FATAL Error: Unknown configuration key(s): "port". Check for spelling errors and ensure that expected plugins are installed.
Jun 14 11:12:09 wazuh-server systemd[1]: kibana.service: Main process exited, code=exited, status=64/USAGE
Jun 14 11:12:09 wazuh-server systemd[1]: kibana.service: Failed with result 'exit-code'.
systemd[1]: Starting Elasticsearch...
systemd-entrypoint[5678]: Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [elasticsear>
systemd-entrypoint[5678]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1193)
systemd-entrypoint[5678]: at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:140)
systemd-entrypoint[5678]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:54)
systemd-entrypoint[5678]: at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
systemd-entrypoint[5678]: at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:86)
systemd-entrypoint[5678]: at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)
systemd-entrypoint[5678]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
systemd-entrypoint[5678]: at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)
systemd-entrypoint[5678]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
systemd-entrypoint[5678]: at org.elasticsearch.cli.Command.main(Command.java:77)
systemd-entrypoint[5678]: at org.elasticsearch.cli.keystore.KeyStoreCli.main(KeyStoreCli.java:33)
systemd-entrypoint[5678]: Caused by: org.elasticsearch.xcontent.XContentParseException: [113:24] Duplicate field 'xpack.security.http.ssl'
systemd-entrypoint[5678]: at [Source: (ByteArrayInputStream); line: 113, column: 24]
systemd-entrypoint[5678]: at org.elasticsearch.xcontent.provider.json.JsonXContentParser.newXContentParseException(JsonXContentParser.java:>
systemd-entrypoint[5678]: at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextToken(JsonXContentParser.java:62)
systemd-entrypoint[5678]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:718)
systemd-entrypoint[5678]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:687)
systemd-entrypoint[5678]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1189)
systemd-entrypoint[5678]: ... 10 more
systemd-entrypoint[5678]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'xpack.security.http.ssl'
systemd-entrypoint[5678]: at [Source: (ByteArrayInputStream); line: 113, column: 24]
systemd-entrypoint[5678]: at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:225)
systemd-entrypoint[5678]: at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:219)
systemd-entrypoint[5678]: at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:456)
systemd-entrypoint[5678]: at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextToken(JsonXContentParser.java:58)
systemd-entrypoint[5678]: ... 13 more
systemd-entrypoint[5748]: Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [elasticsear>
systemd-entrypoint[5748]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1193)
systemd-entrypoint[5748]: at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:140)
systemd-entrypoint[5748]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:54)
systemd-entrypoint[5748]: at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
systemd-entrypoint[5748]: at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:86)
systemd-entrypoint[5748]: at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)
systemd-entrypoint[5748]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
systemd-entrypoint[5748]: at org.elasticsearch.cli.Command.main(Command.java:77)
systemd-entrypoint[5748]: at org.elasticsearch.xpack.security.cli.AutoConfigureNode.main(AutoConfigureNode.java:157)
systemd-entrypoint[5748]: Caused by: org.elasticsearch.xcontent.XContentParseException: [113:24] Duplicate field 'xpack.security.http.ssl'
systemd-entrypoint[5748]: at [Source: (ByteArrayInputStream); line: 113, column: 24]
systemd-entrypoint[5748]: at org.elasticsearch.xcontent.provider.json.JsonXContentParser.newXContentParseException(JsonXContentParser.java:>
systemd-entrypoint[5748]: at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextToken(JsonXContentParser.java:62)
systemd-entrypoint[5748]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:718)
systemd-entrypoint[5748]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:687)
systemd-entrypoint[5748]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1189)
systemd-entrypoint[5748]: ... 8 more
systemd-entrypoint[5748]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'xpack.security.http.ssl'
actually right now i have problem with the elastic don't start any more , i think some thing wrong with the xpack.security, just don't know how i can fix it, if you can help, the last error log o posted for elastic
systemd-entrypoint[5678]: Caused by: org.elasticsearch.xcontent.XContentParseException: [113:24] Duplicate field 'xpack.security.http.ssl'
Looks like a duplicate field... I am not sure... I am not sure what state you are in/ what your configuration is
I would suggest Start Over - completely clean up / uninstall elasticsearch and kibana (including all the install directories /etc directories etc.... and just install from scratch following the exact minimal / defaults.
The ONLY things I changed was 1 line in my kibana.yml before I started kibana was
server.host: "0.0.0.0"
Which allows you to connect to kibana from another host.
I have a complete install up and running using .DEB on ubuntu in less than 10 minutes... with SSL for elastic and authentication
I have no idea all the changes you have made... starting over is my suggestion...
i had every thing working perfect until up yesterday. i came to login in kibana as usual and it was saying cant log you in. i rebooted the server then i find out the problem actually from Elasticsearch not kibana , yesterday i restore the last vmware snapshoot, and asll was workign back to normal and i didnt update ot change any thing at all, but the problem back again on it on.
bellow is my elasticsearch.yml config.
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 14-06-2022 10:35:36
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
## Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
##xpack.security.http.ssl:
## enabled: true
## keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
#keystore.path: certs/transport.p12
keystore.path: certs/http.p12
truststore.path: certs/transport.p12
xpack.security.http.ssl:
enabled: true
key: /etc/elasticsearch/ssl/privkey.pem
certificate: /etc/elasticsearch/ssl/fullchain.pem
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["wazuh-server"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
Apologies... but perhaps someone else can help from here on... it seem that you prefer not to follow my suggestions...
Here is my working elasticsearch.yml with the defaults. my host name is stephenb-es-8-test
cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
# network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 27-06-2022 18:30:48
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["stephenb-es-8-test"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
If you compare the 2 you can see they are close but your is missing certain fields and formatted incorrectly etc looks like you have been trying to edit it..... most likely the reason your elasticsearch is failing.
i will try to reformatted the same way you have yours but the problem when i restore to to previous vmware snapshot all will be working perfect no problem with no errors with the same config file.
It is not just formatting you are missing complete settings...
Restoring snapshots, hand editing the yml when you are unsure of the settings... too many variables... if this does not work ... clean up complete and re-install from scratch ... your spending more time try to "fix" this than just re-install unless you already have a lot of valuable data.
Thank you for your help, i restored the snapshot to the working version, also i fixed the config file as you sent me, stooped kibana & installed it on a stand alone VM all is working good so far. will give it few hours and see if the problem will come back again. hopefully not.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.