Kibana 6.6.2 fails with Unable to revive connection: https://sample-elasticsearch.elasticsearch:9200/

Bharath_venkat · October 22, 2019, 12:12pm

Hi ,
I am trying to create SAML SSO authentication and secure elk with TLS/SSL on kubernetes . Following the post .

Kibana is unable to connect to elasticsearch. I could successfully deploy elasticsearch with x-pack tls/ssl . Querying the https://localhost:9200 gives me

"name" : "nPzJYMW",
"cluster_name" : "sample-elasticsearch-cluster",
"cluster_uuid" : "Ec7pRhtkQKCHOFFBwp5H_w",
"version" : {
"number" : "6.6.2",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "3bd3e59",
"build_date" : "2019-03-06T15:16:26.864148Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

When i try to query https://locahost:5601 for kibana i get "kibana server is not ready". Part of the logs shows as below

{"type":"log","@timestamp":"2019-10-22T11:59:16Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"error","@timestamp":"2019-10-22T11:59:18Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n at TLSSocket.emit (events.js:194:15)\n at _handle.close (net.js:600:12)\n at Socket.done (_tls_wrap.js:388:7)\n at Object.onceWrapper (events.js:277:13)\n at Socket.emit (events.js:189:13)\n at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"error","@timestamp":"2019-10-22T11:59:18Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n at TLSSocket.emit (events.js:194:15)\n at _handle.close (net.js:600:12)\n at Socket.done (_tls_wrap.js:388:7)\n at Object.onceWrapper (events.js:277:13)\n at Socket.emit (events.js:189:13)\n at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"error","@timestamp":"2019-10-22T11:59:18Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n at TLSSocket.emit (events.js:194:15)\n at _handle.close (net.js:600:12)\n at Socket.done (_tls_wrap.js:388:7)\n at Object.onceWrapper (events.js:277:13)\n at Socket.emit (events.js:189:13)\n at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"error","@timestamp":"2019-10-22T11:59:18Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n at TLSSocket.emit (events.js:194:15)\n at _handle.close (net.js:600:12)\n at Socket.done (_tls_wrap.js:388:7)\n at Object.onceWrapper (events.js:277:13)\n at Socket.emit (events.js:189:13)\n at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"log","@timestamp":"2019-10-22T11:59:19Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://sample-elasticsearch:9200/"}
{"type":"log","@timestamp":"2019-10-22T11:59:19Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}

Kibana config :
kibana.yml: |
server.name: sample-kibana
server.host: "0.0.0.0"
elasticsearch.url: https://sample-elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: kibana
elasticsearch.password: password1$
xpack.security.encryptionKey: "bharathkumarkvvenkataramanaiahkvtarba"
elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/elastic-stack-ca.pem"
server.ssl.enabled: true
server.ssl.key: "/usr/share/kibana/config/instance.key"
server.ssl.certificate: "/usr/share/kibana/config/instance.crt"
xpack.monitoring.elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.verificationMode: certificate
xpack.security.authProviders: [saml, basic]
server.xsrf.whitelist: [/api/security/v1/saml]
xpack.security.public:
protocol: https
hostname: saml-aad.elastictest.co

elasticsearch config:
elasticsearch.yml: |
cluster.name: "sample-elasticsearch-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
# Update max_local_storage_nodes value based on number of nodes
node.max_local_storage_nodes: 1
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.license.self_generated.type: basic
xpack.ssl.keystore.type: PKCS12
xpack.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.ssl.keystore.password: password1$
xpack.ssl.truststore.type: PKCS12
xpack.ssl.truststore.path: "/usr/share/elasticsearch/config/elastic-certificates.p12"
xpack.ssl.truststore.password: password1$
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.ssl.verification_mode: certificate
http.cors.enabled: true
http.cors.allow-origin: "*"
http.max_header_size: 16kb
xpack:
security:
authc:
realms:
native1:
type: native
order: 0
saml1:
type: saml
order: 2
idp.metadata.path: "/usr/share/elasticsearch/config/Elasticsearch.xml"
idp.entity_id: "https://sts.windows.net/6d5e78da-01ca-4ff1-82fe-bad7a7e64f96/"
sp.entity_id: "https://saml-aad.elastictest.co:5601"
sp.acs: "https://saml-aad.elastictest.co:5601/api/security/v1/saml"
sp.logout: "https://saml-aad.elastictest.co:5601/logout"
attributes.principal: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"
attributes.groups: "http://schemas.microsoft.com/ws/2008/06/identity/claims/roles"
role_mapping.yml: |

bhavyarm · October 22, 2019, 1:39pm

@azasypkin can you please grab this one?

Thank you
Bhavya

Bharath_venkat · October 23, 2019, 3:33am

Hey guys any update here .

ikakavas · October 23, 2019, 7:50am

Thanks very much for your interest in Elasticsearch.

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question and only 16hrs had passed when you asked for an update. For more information please refer to the Community Code of Conduct specifically the section "Be patient".

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

Also, please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

Bharath_venkat · October 23, 2019, 11:34am

I appreciate your patience to answer the questions . Well had to rewrite the post in a readable format again .

I am trying to create SAML SSO authentication and secure elk with TLS/SSL on kubernetes . Following the post .

Kibana is unable to connect to elasticsearch. I could successfully deploy elasticsearch with x-pack tls/ssl . Querying the https://localhost:9200 gives me

"name" : "nPzJYMW",
"cluster_name" : "sample-elasticsearch-cluster",
"cluster_uuid" : "Ec7pRhtkQKCHOFFBwp5H_w",
"version" : {
"number" : "6.6.2",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "3bd3e59",
"build_date" : "2019-03-06T15:16:26.864148Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

When i try to query https://locahost:5601 for kibana i get "kibana server is not ready yet". Part of the logs shows as below

{"type":"log","@timestamp":"2019-10-23T06:58:36Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"log","@timestamp":"2019-10-23T06:58:38Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://sample-elasticsearch:9200/"}
{"type":"log","@timestamp":"2019-10-23T06:58:38Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"error","@timestamp":"2019-10-23T06:58:39Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n    at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n    at TLSSocket.emit (events.js:194:15)\n    at _handle.close (net.js:600:12)\n    at Socket.done (_tls_wrap.js:388:7)\n    at Object.onceWrapper (events.js:277:13)\n    at Socket.emit (events.js:189:13)\n    at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"error","@timestamp":"2019-10-23T06:58:39Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n    at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n    at TLSSocket.emit (events.js:194:15)\n    at _handle.close (net.js:600:12)\n    at Socket.done (_tls_wrap.js:388:7)\n    at Object.onceWrapper (events.js:277:13)\n    at Socket.emit (events.js:189:13)\n    at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"error","@timestamp":"2019-10-23T06:58:39Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n    at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n    at TLSSocket.emit (events.js:194:15)\n    at _handle.close (net.js:600:12)\n    at Socket.done (_tls_wrap.js:388:7)\n    at Object.onceWrapper (events.js:277:13)\n    at Socket.emit (events.js:189:13)\n    at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"error","@timestamp":"2019-10-23T06:58:39Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n    at TLSSocket.onSocketClose (_tls_wrap.js:761:23)\n    at TLSSocket.emit (events.js:194:15)\n    at _handle.close (net.js:600:12)\n    at Socket.done (_tls_wrap.js:388:7)\n    at Object.onceWrapper (events.js:277:13)\n    at Socket.emit (events.js:189:13)\n    at TCP._handle.close (net.js:600:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"log","@timestamp":"2019-10-23T06:58:41Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: https://sample-elasticsearch:9200/"}
{"type":"log","@timestamp":"2019-10-23T06:58:41Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"log","@timestamp":"2019-10-23T06:58:41Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"Unable to revive connection: https://sample-elasticsearch:9200/"}
{"type":"log","@timestamp":"2019-10-23T06:58:41Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"No living connections"}
{"type":"log","@timestamp":"2019-10-23T06:58:41Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"}
{"type":"log","@timestamp":"2019-10-23T06:58:43Z","tags":["warning","elasticsear

Bharath_venkat · October 23, 2019, 11:34am

Continued

Kibana config:

data:
  kibana.yml: |
    server.name: sample-kibana
    server.host: "0.0.0.0"
    elasticsearch.url: https://sample-elasticsearch:9200
    xpack.monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch.username: kibana
    elasticsearch.password: password1$
    xpack.security.encryptionKey: "bharathkumarkvvenkataramanaiahkvtarba"
    elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/elastic-stack-ca.pem"
    server.ssl.enabled: true
    server.ssl.key: "/usr/share/kibana/config/instance.key"
    server.ssl.certificate: "/usr/share/kibana/config/instance.crt"
    xpack.monitoring.elasticsearch.ssl.verificationMode: certificate
    elasticsearch.ssl.verificationMode: certificate
    xpack.security.authProviders: [saml, basic]
    server.xsrf.whitelist: [/api/security/v1/saml]
    xpack.security.public:
      protocol: https
      hostname: saml-aad.elastictest.co

Elasticsearch config :

elasticsearch.yml: |
    cluster.name: "sample-elasticsearch-cluster"
    network.host: 0.0.0.0
    discovery.zen.minimum_master_nodes: 1
    # Update max_local_storage_nodes value based on number of nodes
    node.max_local_storage_nodes: 1
    xpack.security.enabled: true
    xpack.monitoring.collection.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.license.self_generated.type: basic
    xpack.ssl.keystore.type: PKCS12
    xpack.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
    xpack.ssl.keystore.password: password1$
    xpack.ssl.truststore.type: PKCS12
    xpack.ssl.truststore.path: "/usr/share/elasticsearch/config/elastic-certificates.p12"
    xpack.ssl.truststore.password: password1$
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
    xpack.ssl.verification_mode: certificate
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    http.max_header_size: 16kb
    xpack:
      security:
        authc:
          realms:
            native1:
                type: native
                order: 0
            saml1:
              type: saml
              order: 2
              idp.metadata.path: "/usr/share/elasticsearch/config/Elasticsearch.xml"
              idp.entity_id: "https://sts.windows.net/6d5e78da-01ca-4ff1-82fe-bad7a7e64f96/"
              sp.entity_id:  "https://saml-aad.elastictest.co:5601"
              sp.acs: "https://saml-aad.elastictest.co:5601/api/security/v1/saml"
              sp.logout: "https://saml-aad.elastictest.co:5601/logout"
              attributes.principal: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"
              attributes.groups: "http://schemas.microsoft.com/ws/2008/06/identity/claims/roles"
  role_mapping.yml: |

ikakavas · October 23, 2019, 11:52am

You connect to https://localhost:9200 but you configure kibana to connect to https://sample-elasticsearch:9200, so the fact that you can access the former doesn't help much in this case.

Is sample-elasticsearch on port 9200 resolvable and reachable from within the kibana container ?
Any errors shown in the elasticsearch logs ?

Bharath_venkat · October 23, 2019, 12:35pm

i cannot connect to sample-elasticsearch:9200 from the container

sh-4.2$ curl https://sample-elasticsearch:9200
curl: (6) Could not resolve host: sample-elasticsearch; Unknown error

but elasticsearch response to the ping since kibana and elastic are in same namespace

sh-4.2$ ping 10.160.177.139
PING 10.160.177.139 (10.160.177.139) 56(84) bytes of data.
64 bytes from 10.160.177.139: icmp_seq=1 ttl=64 time=3.82 ms
64 bytes from 10.160.177.139: icmp_seq=2 ttl=64 time=0.893 ms
64 bytes from 10.160.177.139: icmp_seq=3 ttl=64 time=0.789 ms
64 bytes from 10.160.177.139: icmp_seq=4 ttl=64 time=1.15 ms
64 bytes from 10.160.177.139: icmp_seq=5 ttl=64 time=0.774 ms

Elasticsearch has no error logs . I can help you with any logs if required

[2019-10-23T06:51:37,913][INFO ][o.e.d.DiscoveryModule    ] [nPzJYMW] using discovery type [zen] and host providers [settings]
[2019-10-23T06:51:40,118][INFO ][o.e.n.Node               ] [nPzJYMW] initialized
[2019-10-23T06:51:40,119][INFO ][o.e.n.Node               ] [nPzJYMW] starting ...
[2019-10-23T06:51:40,521][INFO ][o.e.t.TransportService   ] [nPzJYMW] publish_address {10.160.177.139:9300}, bound_addresses {0.0.0.0:9300}
[2019-10-23T06:51:40,615][INFO ][o.e.b.BootstrapChecks    ] [nPzJYMW] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-10-23T06:51:43,724][INFO ][o.e.c.s.MasterService    ] [nPzJYMW] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {nPzJYMW}{nPzJYMWjSNqJSYNZBoXiIw}{0ua-YR2TSVmz_kcTMvO8qQ}{10.160.177.139}{10.160.177.139:9300}{ml.machine_memory=14683115520, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-10-23T06:51:43,737][INFO ][o.e.c.s.ClusterApplierService] [nPzJYMW] new_master {nPzJYMW}{nPzJYMWjSNqJSYNZBoXiIw}{0ua-YR2TSVmz_kcTMvO8qQ}{10.160.177.139}{10.160.177.139:9300}{ml.machine_memory=14683115520, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {nPzJYMW}{nPzJYMWjSNqJSYNZBoXiIw}{0ua-YR2TSVmz_kcTMvO8qQ}{10.160.177.139}{10.160.177.139:9300}{ml.machine_memory=14683115520, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-10-23T06:51:44,010][INFO ][o.e.h.n.Netty4HttpServerTransport] [nPzJYMW] publish_address {10.160.177.139:9200}, bound_addresses {0.0.0.0:9200}
[2019-10-23T06:51:44,010][INFO ][o.e.n.Node               ] [nPzJYMW] started
[2019-10-23T06:51:47,412][INFO ][o.e.l.LicenseService     ] [nPzJYMW] license [50fb8149-ed09-4d4e-91de-483e3a9bb4b1] mode [trial] - valid
[2019-10-23T06:51:47,424][INFO ][o.e.g.GatewayService     ] [nPzJYMW] recovered [9] indices into cluster_state
[2019-10-23T06:51:58,822][INFO ][o.e.c.r.a.AllocationService] [nPzJYMW] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.watcher-history-9-2019.10.21][0], [.monitoring-es-6-2019.10.23][0], [.monitoring-es-6-2019.10.21][0], [.monitoring-es-6-2019.10.22][0]] ...]).
[2019-10-23T06:56:20,853][INFO ][o.e.m.j.JvmGcMonitorService] [nPzJYMW] [gc][280] overhead, spent [292ms] collecting in the last [1s]
[2019-10-23T10:57:02,410][INFO ][o.e.m.j.JvmGcMonitorService] [nPzJYMW] [gc][14707] overhead, spent [262ms] collecting in the last [1s]

We have a old ELK setup which is in production . The kibana.yaml has entry for elasticsearch as "http://elasticsearch-svc.elasticsearch.svc.cluster.local:9200" . Strangely kibana.yml is not mounted to the kibana container but still kibana has no issue talking to elasticsearch

ikakavas · October 24, 2019, 5:04am

Ok, so the problem is that you have configured Kibana to connect to https://sample-elasticsearch:9200 but this hostname can't be resolved from within the Kibana container.

I unfortunately cannot help you much with your kubernetes setup, but you need to figure out as which host is elasticsearch available from the kibana container and use that hostname in

elasticsearch.url: https://<the_hostname_here>:9200

As an aside, have you looked into ECK, instead of managing your k8s yourself ?

system · November 21, 2019, 5:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.