Hi,
last week I updated our elasticsearch server, but from then kibana refuses to start.
Here is its log:
Oct 15 12:26:55 sub.server.com systemd[1]: Started Kibana.
Oct 15 12:27:05 sub.server.com kibana[26328]: kibana.keystore located in the data folder is deprecated. Future versions will use the config folder.
Oct 15 12:27:05 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:05Z","tags":["warning","plugins-discovery"],"pid":26328,"message":"Expect plugin \"id\" in camelCase, but found: beats_management"}
Oct 15 12:27:05 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:05Z","tags":["warning","plugins-discovery"],"pid":26328,"message":"Expect plugin \"id\" in camelCase, but found: triggers_actions_ui"}
Oct 15 12:27:23 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:23Z","tags":["info","plugins-service"],"pid":26328,"message":"Plugin \"auditTrail\" is disabled."}
Oct 15 12:27:23 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:23Z","tags":["info","plugins-service"],"pid":26328,"message":"Plugin \"visTypeXy\" is disabled."}
Oct 15 12:27:24 sub.server.com kibana[26328]: kibana.keystore located in the data folder is deprecated. Future versions will use the config folder.
Oct 15 12:27:24 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:24Z","tags":["warning","config","deprecation"],"pid":26328,"message":"Setting [elasticsearch.username] to \"kibana\" is deprecated. You should use the \"kibana_system\" user instead."}
Oct 15 12:27:24 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:24Z","tags":["warning","config","deprecation"],"pid":26328,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
Oct 15 12:27:24 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:24Z","tags":["warning","config","deprecation"],"pid":26328,"message":"Setting [monitoring.username] to \"kibana\" is deprecated. You should use the \"kibana_system\" user instead."}
Oct 15 12:27:24 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:24Z","tags":["info","plugins-system"],"pid":26328,"message":"Setting up [92] plugins: [usageCollection,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,newsfeed,mapsLegacy,kibanaLegacy,taskManager,ossTelemetry,licensing,observability,globalSearch,globalSearchProviders,code,timelion,share,legacyExport,esUiShared,charts,bfetch,expressions,data,home,console,consoleExtensions,apmOss,cloud,management,upgradeAssistant,licenseManagement,indexPatternManagement,advancedSettings,watcher,searchprofiler,painlessLab,grokdebugger,savedObjects,visualizations,visualize,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,features,security,snapshotRestore,reporting,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,beats_management,transform,ingestPipelines,graph,canvas,visTypeMarkdown,visTypeTagcloud,visTypeMetric,visTypeTable,tileMap,regionMap,inputControlVis,discover,discoverEnhanced,dashboard,lens,dashboardMode,savedObjectsManagement,spaces,lists,eventLog,actions,case,alerts,alertingBuiltins,ml,apm,uptime,fileUpload,maps,dataEnhanced,securitySolution,infra,monitoring,logstash,translations]"}
Oct 15 12:27:25 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:25Z","tags":["info","plugins","monitoring","monitoring"],"pid":26328,"message":"config sourced from: production cluster"}
Oct 15 12:27:25 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:25Z","tags":["info","plugins","reporting","config"],"pid":26328,"message":"Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu Linux 18.04 OS. Automatically enabling Chromium sandbox."}
Oct 15 12:27:25 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:25Z","tags":["warning","plugins","licensing"],"pid":26328,"message":"License information could not be obtained from Elasticsearch due to [security_exception] unable to authenticate user [kibana] for REST request [/_xpack], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" & 1=\"Bearer realm=\\\"security\\\"\" & 2=\"ApiKey\" } } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [kibana] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":[\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\",\\\"Bearer realm=\\\\\\\"security\\\\\\\"\\\",\\\"ApiKey\\\"]}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [kibana] for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":[\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\",\\\"Bearer realm=\\\\\\\"security\\\\\\\"\\\",\\\"ApiKey\\\"]}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\", Bearer realm=\\\"security\\\", ApiKey\"} error"}
Oct 15 12:27:25 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:25Z","tags":["warning","plugins","monitoring","monitoring"],"pid":26328,"message":"X-Pack Monitoring Cluster Alerts will not be available: [security_exception] unable to authenticate user [kibana] for REST request [/_xpack], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" & 1=\"Bearer realm=\\\"security\\\"\" & 2=\"ApiKey\" } } }"}
Oct 15 12:27:25 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:25Z","tags":["fatal","root"],"pid":26328,"message":"URIError: URI malformed\n at decodeURIComponent (<anonymous>)\n at getUsernameAndPassword (/usr/share/kibana/node_modules/@elastic/elasticsearch/index.js:251:19)\n at getAuth (/usr/share/kibana/node_modules/@elastic/elasticsearch/index.js:224:20)\n at new Client (/usr/share/kibana/node_modules/@elastic/elasticsearch/index.js:59:23)\n at configureClient (/usr/share/kibana/src/core/server/elasticsearch/client/configure_client.js:37:18)\n at new ClusterClient (/usr/share/kibana/src/core/server/elasticsearch/client/cluster_client.js:40:65)\n at ElasticsearchService.createClusterClient (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:135:12)\n at ElasticsearchService.start (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:104:24)\n at process._tickCallback (internal/process/next_tick.js:68:7)"}
Oct 15 12:27:25 sub.server.com kibana[26328]: {"type":"log","@timestamp":"2020-10-15T10:27:25Z","tags":["info","plugins-system"],"pid":26328,"message":"Stopping all plugins."}
Oct 15 12:27:25 sub.server.com kibana[26328]: FATAL URIError: URI malformed
I tried changing kibana username to kibana_system and changing its password, but without any results. Here is kibana configuration:
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: ""
#server.basePath: ""
#server.rewriteBasePath: false
#server.maxPayloadBytes: 1048576
server.name: "serverName"
elasticsearch.hosts: ["https://localhost:9200"]
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"
# The default application to load.
#kibana.defaultAppId: "home"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibana_password"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /path/to/cert.pem
server.ssl.key: /path/to/privkey.pem
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
elasticsearch.ssl.verificationMode: none
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000
# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false
# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
server.host: "0.0.0.0"
xpack.encryptedSavedObjects.encryptionKey: "longenoughkey"
xpack.reporting.encryptionKey: "longenoughkey"
xpack.security.encryptionKey: "longenoughkey"
I tried the following commands and they are successful:
curl -k -s -u kibana_system:kibanapassword https://localhost:9200/_xpack/security/_authenticate?pretty
curl -k -s -u kibana:kibanapassword https https://localhost:9200/_xpack/security/_authenticate?pretty
{
"username" : "kibana_system",
"roles" : [
"kibana_system"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true,
"authentication_realm" : {
"name" : "reserved",
"type" : "reserved"
},
"lookup_realm" : {
"name" : "reserved",
"type" : "reserved"
}
}
The password does not contain any "%".
I cannot understand what the problem could be.