Kibana unable to connect to elasticsearch after upgrade 6.6.8 to 7.5.0

Elastic Stack Elasticsearch
1

Using docker here. Upgraded elasticsearch and now cannot attach to it with kibana. We have a premium license and have had x-pack configured. I inherited it and honestly do not understand the configuration all that well. I think my settings need to change. Is there a way to start from scratch in the configuration?

The elastic cluster is up and running:

[root@elasticsearch etc]#  curl -k -u elastic -XGET 'https://localhost/_cluster/health?pretty'
Enter host password for user 'elastic':
{
  "cluster_name" : "logging",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 925,
  "active_shards" : 925,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Log file for elasticsearch:
{"type": "server", "timestamp": "2019-12-13T15:06:10,662Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "logging", "node.name": "elasticsearch", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2019-12-13T15:06:14,241Z", "level": "INFO", "component": "o.e.x.s.a.l.LdapUserSearchSessionFactory", "cluster.name": "logging", "node.name": "elasticsearch", "message": "Realm [ldap1] is in user-search mode - base_dn=[ou=Users,o=54899d1f318ab54f7100d8f0,dc=jumpcloud,dc=com], search filter=[(uid={0})]" }
{"type": "server", "timestamp": "2019-12-13T15:06:14,279Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "logging", "node.name": "elasticsearch", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2019-12-13T15:06:14,797Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "logging", "node.name": "elasticsearch", "message": "[controller/263] [Main.cc@110] controller (64 bit): Version 7.5.0 (Build 17d1c724ca38a1) Copyright (c) 2019 Elasticsearch BV" }
{"type": "server", "timestamp": "2019-12-13T15:06:15,344Z", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "logging", "node.name": "elasticsearch", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
{"type": "server", "timestamp": "2019-12-13T15:06:15,499Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "logging", "node.name": "elasticsearch", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2019-12-13T15:06:16,471Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "logging", "node.name": "elasticsearch", "message": "initialized" }
{"type": "server", "timestamp": "2019-12-13T15:06:16,471Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "logging", "node.name": "elasticsearch", "message": "starting ..." }
{"type": "server", "timestamp": "2019-12-13T15:06:16,603Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "logging", "node.name": "elasticsearch", "message": "publish_address {192.168.80.2:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2019-12-13T15:06:17,443Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "logging", "node.name": "elasticsearch", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2019-12-13T15:06:17,448Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "logging", "node.name": "elasticsearch", "message": "cluster UUID [AQkjVSEAR9i2ItC6QVghow]" }
{"type": "server", "timestamp": "2019-12-13T15:06:17,642Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "logging", "node.name": "elasticsearch", "message": "elected-as-master ([1] nodes joined)[{elasticsearch}{05w2lDl6RMyhX60gts6pFw}{_0AwjYJSTOeM6eCsjS2GQw}{192.168.80.2}{192.168.80.2:9300}{dilm}{ml.machine_memory=59087360000, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 5, version: 2700, delta: master node changed {previous [], current [{elasticsearch}{05w2lDl6RMyhX60gts6pFw}{_0AwjYJSTOeM6eCsjS2GQw}{192.168.80.2}{192.168.80.2:9300}{dilm}{ml.machine_memory=59087360000, xpack.installed=true, ml.max_open_jobs=20}]}" }
{"type": "server", "timestamp": "2019-12-13T15:06:17,696Z", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "logging", "node.name": "elasticsearch", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:55634}], closing connection",
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No available authentication scheme",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:473) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: javax.net.ssl.SSLHandshakeException: No available authentication scheme",
"at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
"at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:258) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateProducer.onProduceCertificate(CertificateMessage.java:951) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateProducer.produce(CertificateMessage.java:940) ~[?:?]",
"at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440) ~[?:?]",
"at sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1243) ~[?:?]",
"at sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1179) ~[?:?]",
"at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:851) ~[?:?]",
"at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:812) ~[?:?]",
"at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]",
"at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]",
"at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]",
"... 16 more"] }
{
2

config file:

cluster.name                       : logging
network.host                       : 0.0.0.0
#discovery.zen.minimum_master_nodes : 1
#xpack.license.self_generated.type  : trial
#discovery.zen.ping.unicast.hosts: ["0.0.0.0"]
node.name: "elasticsearch"
cluster.initial_master_nodes: ["elasticsearch"]
# Added repo for snapshots 2019/12/04
path.repo: ["/usr/share/elasticsearch/backup"]
# changes for 7.5.0
node.master: true
node.data: true
xpack.security.enabled: true

DO i need to make changes to these settings???

xpack.security.http.ssl.enabled : true
xpack.security.http.ssl.certificate :  /usr/share/elasticsearch/config/certs/service.crt
xpack.security.http.ssl.key :  /usr/share/elasticsearch/config/certs/service.key
xpack.security.transport.ssl.enabled : true
xpack.security.transport.ssl.verification_mode : certificate
xpack.security.transport.ssl.certificate_authorities : /usr/share/elasticsearch/config/certs/ca.crt

xpack:
  security:
    authc:
      realms:
        ldap.ldap1:
          order         : 0
          url           : "ldaps://ldap.jumpcloud.com:636"
          bind_dn       : "uid=application.bind,ou=Users,o=54899d1f318ab54f7100d8f0,dc=jumpcloud,dc=com"
#          bind_password : "<pwd>"
          user_search :
            base_dn : "ou=Users,o=54899d1f318ab54f7100d8f0,dc=jumpcloud,dc=com"
          group_search :
            base_dn : "ou=Users,o=54899d1f318ab54f7100d8f0,dc=jumpcloud,dc=com"
          files:
            role_mapping : /usr/share/elasticsearch/config/x-pack/role_mapping.yml
          unmapped_groups_as_roles : false
          ssl.verification_mode    : none
3

Here is the kibana log:

kibana_1  | {"type":"log","@timestamp":"2019-12-13T15:48:42Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/visualizations"}
kibana_1  | {"type":"log","@timestamp":"2019-12-13T15:48:43Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]"}
kibana_1  | {"type":"log","@timestamp":"2019-12-13T15:48:43Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
kibana_1  | {"type":"log","@timestamp":"2019-12-13T15:48:43Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":[\\\"Bearer realm=\\\\\\\"security\\\\\\\"\\\",\\\"ApiKey\\\",\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"]}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":[\\\"Bearer realm=\\\\\\\"security\\\\\\\"\\\",\\\"ApiKey\\\",\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"]}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Bearer realm=\\\"security\\\", ApiKey, Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}
4

Anyone?

5

start with here. I don't have this setup. I use standard setup with basic license.

curl -k -u kibana -XGET 'https://localhost/_cluster/health?pretty'

From kibana log it clearly says it can't authenticate user kibana

6

Thank you helping!

So as you suspecting the kibana password is no longer valid - not sure how that was changed.

# curl -k -u kibana -XGET 'https://localhost/_cluster/health?pretty'
Enter host password for user 'kibana':
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "failed to authenticate user [kibana]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "failed to authenticate user [kibana]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

I tried to change the password but that didn't work either...

curl -k -u elastic -X PUT "https://localhost/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "s3cr3t"
}
'
Enter host password for user 'elastic':
{
  "error" : {
    "root_cause" : [
      {
        "type" : "action_request_validation_exception",
        "reason" : "Validation Failed: 1: no settings to update;"
      }
    ],
    "type" : "action_request_validation_exception",
    "reason" : "Validation Failed: 1: no settings to update;"
  },
  "status" : 400
}
7

I think you have to use this to change kibana password

curl -u elastic -XPUT 'http://localhost:9200/_xpack/security/user/kibana/_password?pretty' -H 'Content-Type: application/json' -d'
{
"password" : "new_password"
}
'

8

I see my horrendous cut and paste error.

But still there seems to be a problem.

curl -u elastic -XPUT 'http://localhost:9200/_xpack/security/user/kibana/_password?pretty' -H 'Content-Type: application/json' -d'
> {
> "password" : "new_password"
> }
> '
Enter host password for user 'elastic':
curl: (7) Failed connect to localhost:9200; Connection refused
[root@elasticsearch etc]# curl -u elastic -XPUT 'https://localhost:9200/_xpack/security/user/kibana/_password?pretty' -H 'Content-Type: application/json' -d'
{
"password" : "new_password"
}
'
Enter host password for user 'elastic':
curl: (7) Failed connect to localhost:9200; Connection refused
[root@elasticsearch etc]# curl -u elastic -XPUT 'https://localhost/_xpack/security/user/kibana/_password?pretty' -H 'Content-Type: application/json' -d'
{
"password" : "new_password"
}
'
Enter host password for user 'elastic':
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
9

According to logs it is the issue of authentication. So to change the kibana user password use:

curl -k -u elastic -X POST "https://localhost/_security/user/kibana/_password" -d '{"password" : "s3cr3t"}'

After this you need to add/change as below in kibana.yml:

elasticsearch.username: "kibana"
elasticsearch.password: "s3cr3t"

If you don't want to store password in kibana.yml file, you can instead use commands to create a keystore and store user and password there. Follow this guide on how to create and store passwords in keystore.

Once done restart kibana.

10

Here is the line that worked! thank you!

curl -k -u elastic -XPOST 'https://localhost/_security/user/kibana/_password' -H 'Content-Type: application/json' -d'
{
"password" : "s3cr3t"
}
'

This issue has been resolved now. Thanks!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.