Kibana 6.3.0 failing after TLS setup

Elastic Stack Kibana
1

Hi,

I'm in the process of setting up TLS for my ESCluster by following the steps mentioned here: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/configuring-tls.html

ES master and slaves are ok as evidence that I can successfully curl our escluster:

{
  "name" : "esmaster0",
  "cluster_name" : "es_cluster",
  "cluster_uuid" : "1234567asdfghjm",
  "version" : {
    "number" : "6.3.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "424e937",
    "build_date" : "2018-06-11T23:38:03.357887Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

However, Kibana won't start and keeps on crashing.

Here's the error message:

FATAL { ValidationError: child "xpack" fails because [child "security" fails because ["transport" is not allowed, "http" is not allowed]]
    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:181:19)
    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/any.js:651:31)
    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:121:23)
    at Config._commit (/usr/share/kibana/src/server/config/config.js:119:35)
    at Config.set (/usr/share/kibana/src/server/config/config.js:89:10)
    at Config.extendSchema (/usr/share/kibana/src/server/config/config.js:62:10)
    at extendConfigService (/usr/share/kibana/src/plugin_discovery/plugin_config/extend_config_service.js:26:10)
    at <anonymous>
  isJoi: true,
  name: 'ValidationError',
  details:
   [ { message: '"transport" is not allowed',
       path: 'xpack.security.transport',
       type: 'object.allowUnknown',
       context: [Object] },
     { message: '"http" is not allowed',
       path: 'xpack.security.http',
       type: 'object.allowUnknown',
       context: [Object] } ],
  _object:
   { pkg:
      { version: '6.3.0',
        branch: '6.3',
        buildNum: 17230,
        buildSha: 'c360fcacca5594000a2f7c80fe2dcaad5c7ee52c' },
     dev: { basePathProxyTarget: 5603 },
     pid: { exclusive: false },
     cpu: undefined,
     cpuacct: undefined,
     server:
      { port: 5601,
        host: '0.0.0.0',
        name: 'kibana0',
        ssl: [Object],
        maxPayloadBytes: 1048576,
        autoListen: true,
        defaultRoute: '/app/kibana',
        basePath: '',
        rewriteBasePath: false,
        customResponseHeaders: {},
        cors: false,
        xsrf: [Object] },
     logging:
      { dest: '/var/log/kibana.log',
        verbose: true,
        silent: false,
        quiet: false,
        events: {},
        filter: {},
        json: true,
        useUTC: true },
     ops: { interval: 5000 },
     plugins: { scanDirs: [Array], paths: [Array], initialize: true },
     path: { data: '/var/lib/kibana' },
     optimize:
      { enabled: true,
        bundleFilter: '!tests',
        bundleDir: '/usr/share/kibana/optimize/bundles',
        viewCaching: true,
        watch: false,
        watchPort: 5602,
        watchHost: 'localhost',
        watchPrebuild: false,
        watchProxyTimeout: 300000,
        useBundleCache: true,
        profile: false },
     status: { allowAnonymous: false },
     map:
      { manifestServiceUrl: 'https://catalogue.maps.elastic.co/v2/manifest',
        includeElasticMapsService: true },
     tilemap: { options: [Object] },
     regionmap: { includeElasticMapsService: true },
     i18n: { defaultLocale: 'en' },
     xpack:
      { searchprofiler: [Object],
        ml: [Object],
        tilemap: [Object],
        watcher: [Object],
        xpack_main: [Object],
        graph: [Object],
        monitoring: [Object],
        security: [Object] },
     license_management: { enabled: true },
     index_management: { enabled: true },
     input_control_vis: { enabled: true },
     kbn_doc_views: { enabled: true },
     kbn_vislib_vis_types: { enabled: true },
     markdown_vis: { enabled: true },
     spy_modes: { enabled: true },
     state_session_storage_redirect: { enabled: true },
     status_page: { enabled: true },
     table_vis: { enabled: true },
     tagcloud: { enabled: true },
     tile_map: { enabled: true },
     timelion: { enabled: true },
     metric_vis: { enabled: true },
     region_map: { enabled: true } },
  annotate: [Function] }

Here's my kibana.yml:

server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana0"
elasticsearch.url: "https://esmaster0:9200"
elasticsearch.username: "******"
elasticsearch.password: "*****"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/cert/kibana0/kibana0.crt
server.ssl.key: /etc/kibana/cert/kibana0/kibana0.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/cert/ca/ca.crt" ]
logging.dest: /var/log/kibana.log
logging.verbose: true
xpack.graph.enabled: true
xpack.ml.enabled: true
xpack.monitoring.enabled: true
# xpack.reporting.enabled: true
xpack.security.enabled: true
xpack.watcher.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/kibana/cert/kibana0/kibana0.key
xpack.security.transport.ssl.certificate: /etc/kibana/cert/kibana0/kibana0.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/kibana/cert/ca/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/kibana/cert/kibana0/kibana0.key
xpack.security.http.ssl.certificate: /etc/kibana/cert/kibana0/kibana0.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/kibana/cert/ca/ca.crt" ]

Am I missing a configuration or have setup an attribute wrong? I'm lost on why it says I'm using http when I've set everything to use https.

Any of you guys have an idea what's wrong? :slight_smile:

2

You have copied Elasticsearch/X-Pack settings into your Kibana configuration file, but those settings don't have any meaning in Kibana, hence:

child "xpack" fails because [child "security" fails because ["transport" is not allowed, "http" is not allowed]

Could you explain why you got the impression that you should do this? It happes with some frequency, so I suspect there is a gap in our documentation, but I haven't been able to work out what it is that's sending people in the wrong direction.

For setting up TLS across components in the stack, see these instructions (Kibana is step 4)

3

It says in the documentation to copy all the .yml config on all "nodes", so I assumed kibana is part of those "nodes". Will try the setup for kibana in the link you provided

4

Hi Tim,

Followed the setup from your link:

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "kibana0"

# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "https://esmaster0:9200"

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "discover"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "*****"
elasticsearch.password: "*****"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/cert/kibana0/kibana0.crt
server.ssl.key: /etc/kibana/cert/kibana0/kibana0.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/cert/ca/ca.crt" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 0

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
logging.dest: /var/log/kibana.log

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: true

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# The default locale. This locale can be used in certain circumstances to substitute any missing
# translations.
#i18n.defaultLocale: "en"
xpack.graph.enabled: true
xpack.ml.enabled: true
xpack.monitoring.enabled: true
# xpack.reporting.enabled: true
xpack.security.enabled: true
xpack.watcher.enabled: true
xpack.security.encryptionKey: "**********************"

and restarted kibana using: systemctl restart kibana
even using: systemctl stop kibana and systemctl start kibana

But Kibana is still crashing with this error:

FATAL { ValidationError: child "xpack" fails because [child "security" fails because ["transport" is not allowed, "http" is not allowed]]
        at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:181:19)
        at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/any.js:651:31)
        at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:121:23)
        at Config._commit (/usr/share/kibana/src/server/config/config.js:119:35)
        at Config.set (/usr/share/kibana/src/server/config/config.js:89:10)
        at Config.extendSchema (/usr/share/kibana/src/server/config/config.js:62:10)
        at extendConfigService (/usr/share/kibana/src/plugin_discovery/plugin_config/extend_config_service.js:26:10)
        at <anonymous>

Does kibana "cache" configuration?

5

Still stuck in this :frowning: I've restored my kibana.yml to its old state. No trace of X-pack, but still that error is being thrown.

Any ideas if its config caching or other files that I should configure/update?

6

Kibana shouldn't be caching anything.

Starting with the obvious things to check:

  1. Are you sure that error is recent? Is it possible that you're just seeing the old error message?
  2. Are you sure that you're editing the right config file?
7

Found the issue with:

ValidationError: child "xpack" fails because [child "security" fails because ["transport" is not allowed, "http" is not allowed]]

we've ran these commands in our kibana vm due to the confusion from above issue:

bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password

and

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

Kibana then fails on start because as Tim have mentioned, transport and ssl since those settings don't mean anything in kibana.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.