Good afternoon! Has anyone come across this issue? When xpack is enabled, I get an error, but with xpack disabled, I can use the console without any problems. The certificates are valid, and the login credentials for kibana_user are correct. Any ideas on what might be causing this?
Could you please confirm if you are passing the user id / password in the kibana.yml & that credentials are working?
Also try once using elastic userid/password in kibana.yml to understand if it is user issue or authentication issue.
Welcome to the community, please don't post images of text, they hard to read and can not be searched or debugged.
Quick point of clarification:
That will not work in 8.X the elastic user is not allowed as a user to connect to kibana server in 8.x only kibana_system , the elastic user will always fail (that changed from 7.x to 8.x)
kibana_system not kibana_user
You are asking for full client verificationMode but I do not see anywhere you reference the Elasticsearch CA. not sure if that is the whole kibana.yml but you need to probably at the HTTP CA
elasticsearch.ssl.certificateAuthorities
Good afternoon, thanks for the answer, here is the full config
cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: here host
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: ["10.33.1.80", "127.0.0.1"]
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: []
discovery.type: single-node
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: https.p12
cat /etc/kibana/kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
#server.port: 443
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "10.33.1.80"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
xpack.security.encryptionKey: "tsb2N1IPZO0MBoyCB8LAubhlmDBA1YsMLNkdr2c/ZWk="
xpack.reporting.encryptionKey: "tmwbH+G+Oug/gDs3q5kFfYONBGqBMoi4SiDk50p0/jM="
xpack.encryptedSavedObjects.encryptionKey: "khcnCdCxCvaNtFoTU0v6s1JpEy4xJR3scbm+Q/HiZ/0="
elasticsearch.ssl.verificationMode: full
#xpack.security.enabled: true
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: "http://10.33.1.80:5601"
server.publicBaseUrl: "https://domain:5601"
# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://10.33.1.80:9200"]
elasticsearch.hosts: ["https://domain:9200"]
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"
# The default application to load.
#kibana.defaultAppId: "home"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana_system"
elasticsearch.password: "passwd"
# Kibana can also authenticate to Elasticsearch via "service account tokens".
# If may use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/SSL certificate for Kibana
server.ssl.key: /etc/kibana/the private key associated with the certificate
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
elasticsearch.ssl.certificateAuthorities: /etc/kibana/ca-chain.pem
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 120000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid
# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
Test Connection Using cURL
root@kir-wazuh01:~# curl -u "kibana_system:passwd" -X GET "https://domain:9200" --cacert /etc/kibana/ca-chain.pem
{
"name" : "node-1",
"cluster_name" : "name cluster",
"cluster_uuid" : "",
"version" : {
"number" : "7.17.5",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "",
"build_date" : "2022-06-23T21:57:28.736740635Z",
"build_snapshot" : false,
"lucene_version" : "8.11.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
tail -f /var/log/kibana/kibana.log
{"type":"log","@timestamp":"2024-11-11T18:12:41+03:00","tags":["warning","config","deprecation"],"pid":2162,"message":"User sessions will automatically time out after 8 hours of inactivity starting in 8.0. Override this value to change the timeout."}
{"type":"log","@timestamp":"2024-11-11T18:12:41+03:00","tags":["warning","config","deprecation"],"pid":2162,"message":"Users are automatically required to log in again after 30 days starting in 8.0. Override this value to change the timeout."}
{"type":"log","@timestamp":"2024-11-11T18:12:41+03:00","tags":["info","plugins-system","standard"],"pid":2162,"message":"Setting up [114] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,wazuh,indexPatternManagement]"}
{"type":"log","@timestamp":"2024-11-11T18:12:42+03:00","tags":["info","plugins","taskManager"],"pid":2162,"message":"TaskManager is identified by the Kibana UUID: 3197b8ae-b888-44bf-a117-faca6068aaa3"}
{"type":"log","@timestamp":"2024-11-11T18:12:42+03:00","tags":["info","plugins","ruleRegistry"],"pid":2162,"message":"Installing common resources shared between all indices"}
{"type":"log","@timestamp":"2024-11-11T18:12:42+03:00","tags":["warning","plugins","reporting","config"],"pid":2162,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Debian 11 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2024-11-11T18:12:42+03:00","tags":["warning","process"],"pid":2162,"message":"Error [ProductNotSupportedSecurityError]: The client is unable to verify that the server is Elasticsearch due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.\n at /usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:576:19\n at onBody (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:369:9)\n at IncomingMessage.onEnd (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)\n at IncomingMessage.emit (node:events:538:35)\n at endReadableNT (node:internal/streams/readable:1345:12)\n at processTicksAndRejections (node:internal/process/task_queues:83:21)"}
{"type":"log","@timestamp":"2024-11-11T18:12:42+03:00","tags":["error","elasticsearch-service"],"pid":2162,"message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
{"type":"log","@timestamp":"2024-11-11T18:32:42+03:00","tags":["error","plugins","ruleRegistry"],"pid":2162,"message":"Error: Timeout: it took more than 1200000ms\n at Timeout._onTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:54:20)\n at listOnTimeout (node:internal/timers:559:17)\n at processTimers (node:internal/timers:502:7)"}
{"type":"log","@timestamp":"2024-11-11T18:32:42+03:00","tags":["error","plugins","ruleRegistry"],"pid":2162,"message":"Error: Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms\n at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:63:13)\n at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:77:5)"}
# curl -u "kibana_system:passwd" -X GET "https://10.33.1.80:9200" -k -
also passes
openssl verify -CAfile /etc/kibana/ca-chain.pem /etc/elasticsearch/server_certificate.pem
/etc/elasticsearch/server_certificate.pem: OK
But if you turn off xpack checking, everything loads and works without errors
It also says in the logs that the password doesnβt work, although when checking everything works
tail -f /var/log/elasticsearch/wazuh.log
[2024-11-12T09:28:09,040][INFO ][o.e.x.s.a.RealmsAuthenticator] [node-1] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2024-11-12T09:28:11,539][INFO ][o.e.x.s.a.RealmsAuthenticator] [node-1] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2024-11-12T09:28:14,042][INFO ][o.e.x.s.a.RealmsAuthenticator] [node-1] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2024-11-12T09:28:16,540][INFO ][o.e.x.s.a.RealmsAuthenticator] [node-1] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
I fixed your code formatting please put 3 back ticks ``` before and after your code.
2 things to try
Proper syntax
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca-chain.pem" ]
Or try
elasticsearch.ssl.verificationMode: none
See if either of those work
Are there any other errors before the authentication error?
I see you are using 7.17 elasticsearch... What version of Kibana?
And in 7.17 you can use the elastic user
If you are just starting why 7.17 in stead of 8.15?
solved the problem
cd /usr/share/kibana/bin/
./kibana-keystore - run the command and indicate the current password and elastik logs if you have changed the password
./kibana-keystore add elasticsearch.username - new user
./kibana-keystore add elasticsearch.password - new password
1 Like