Elasticsearch cluster is down

yaminisridaran · November 14, 2019, 2:59pm

We have deployed elasticsearch cluster on kubernetes cluster.
We have deployed 3 master nodes, 3 client nodes and 5 data nodes.
We are able to ping the endpoint ips mentioned in the logs from the master,client and data nodes, but the logs are showing connection issue (manual ping works inside the containers).

Master node logs:
[2019-11-14T12:40:57,348][WARN ][o.e.c.NodeConnectionsService] [master-fed-elastic-0] failed to connect to node

{client-fed-elastic-1}{X3EsRyYLScSEaumLVWGlDQ}{sXzX4oU_TzCaJPRnkUEnTg}{192.168.186.209}{192.168.186.209:9300} (tried [1] times)
Client node logs:
2019-11-14T12:36:00,265][WARN ][o.e.c.NodeConnectionsService] [client-fed-elastic-0] failed to connect to node {client-fed-elastic-1} {X3EsRyYLScSEaumLVWGlDQ} {9uHMXe8lTrqsICoT4HyIPQ} {192.168.186.209} {192.168.186.209:9300}

(tried [1] times)
[2019-11-14T12:37:28,771][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [client-fed-elastic-0] SSL Problem Received close_notify during handshake
javax.net.ssl.SSLException: Received close_notify during handshake

Data node logs:
[2019-11-14T12:40:45,109][WARN ][c.f.s.s.t.SearchGuardSSLNettyTransport] [data-fed-elastic-0] send message failed [channel: NettyTcpChannel

{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/192.168.124.43:42068}

]

Can you please help us?

A_B · November 14, 2019, 4:02pm

Hi @yaminisridaran,

could you share your Elasticsearch config? Please share as Preformated text, then it is much easier to read

yaminisridaran · November 15, 2019, 8:49am

My Elasticsearch cluster config:

Master node config:

bootstrap:
  memory_lock: false
cluster:
  name: ${CLUSTER_NAME}
discovery:
  zen:
    minimum_master_nodes: ${NUMBER_OF_MASTERS}
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
http:
  compression: true
  cors:
    allow-credentials: true
    allow-headers: X-Requested-With, Content-Type, Content-Length, Authorization
    allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
    enabled: ${HTTP_CORS_ENABLE}
  enabled: ${HTTP_ENABLE}
network.host: 0.0.0.0
node:
  data: ${NODE_DATA}
  ingest: ${NODE_INGEST}
  master: ${NODE_MASTER}
  name: ${NODE_NAME}
path:
  data: /data/data
  logs: /data/log
searchguard.authcz.admin_dn:
- CN=sgadmin,O=Elasticsearch Operator
searchguard.disabled: ${SEARCHGUARD_DISABLED:false}
searchguard.enterprise_modules_enabled: false
searchguard.restapi.roles_enabled:
- sg_all_access
- sg_kibana_user
searchguard.ssl.http.enabled: ${SSL_ENABLE}
searchguard.ssl.http.keystore_filepath: certs/client.jks
searchguard.ssl.http.keystore_password: ${KEY_PASS}
searchguard.ssl.http.truststore_filepath: certs/root.jks
searchguard.ssl.http.truststore_password: ${KEY_PASS}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: certs/node.jks
searchguard.ssl.transport.keystore_password: ${KEY_PASS}
searchguard.ssl.transport.truststore_filepath: certs/root.jks
searchguard.ssl.transport.truststore_password: ${KEY_PASS}
thread_pool:
  bulk:
    queue_size: 10000

Client node config:

bootstrap:
  memory_lock: false
cluster:
  name: ${CLUSTER_NAME}
discovery:
  zen:
    minimum_master_nodes: ${NUMBER_OF_MASTERS}
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
http:
  compression: true
  cors:
    allow-credentials: true
    allow-headers: X-Requested-With, Content-Type, Content-Length, Authorization
    allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
    enabled: ${HTTP_CORS_ENABLE}
  enabled: ${HTTP_ENABLE}
network.host: 0.0.0.0
node:
  data: ${NODE_DATA}
  ingest: ${NODE_INGEST}
  master: ${NODE_MASTER}
  name: ${NODE_NAME}
path:
  data: /data/data
  logs: /data/log
searchguard.authcz.admin_dn:
- CN=sgadmin,O=Elasticsearch Operator
searchguard.disabled: ${SEARCHGUARD_DISABLED:false}
searchguard.enterprise_modules_enabled: false
searchguard.restapi.roles_enabled:
- sg_all_access
- sg_kibana_user
searchguard.ssl.http.enabled: ${SSL_ENABLE}
searchguard.ssl.http.keystore_filepath: certs/client.jks
searchguard.ssl.http.keystore_password: ${KEY_PASS}
searchguard.ssl.http.truststore_filepath: certs/root.jks
searchguard.ssl.http.truststore_password: ${KEY_PASS}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: certs/node.jks
searchguard.ssl.transport.keystore_password: ${KEY_PASS}
searchguard.ssl.transport.truststore_filepath: certs/root.jks
searchguard.ssl.transport.truststore_password: ${KEY_PASS}
thread_pool:
  bulk:
    queue_size: 10000

Data node config:

bootstrap:
  memory_lock: false
cluster:
  name: ${CLUSTER_NAME}
discovery:
  zen:
    minimum_master_nodes: ${NUMBER_OF_MASTERS}
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
http:
  compression: true
  cors:
    allow-credentials: true
    allow-headers: X-Requested-With, Content-Type, Content-Length, Authorization
    allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
    enabled: ${HTTP_CORS_ENABLE}
  enabled: ${HTTP_ENABLE}
network.host: 0.0.0.0
node:
  data: ${NODE_DATA}
  ingest: ${NODE_INGEST}
  master: ${NODE_MASTER}
  name: ${NODE_NAME}
path:
  data: /data/data
  logs: /data/log
searchguard.authcz.admin_dn:
- CN=sgadmin,O=Elasticsearch Operator
searchguard.disabled: ${SEARCHGUARD_DISABLED:false}
searchguard.enterprise_modules_enabled: false
searchguard.restapi.roles_enabled:
- sg_all_access
- sg_kibana_user
searchguard.ssl.http.enabled: ${SSL_ENABLE}
searchguard.ssl.http.keystore_filepath: certs/client.jks
searchguard.ssl.http.keystore_password: ${KEY_PASS}
searchguard.ssl.http.truststore_filepath: certs/root.jks
searchguard.ssl.http.truststore_password: ${KEY_PASS}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: certs/node.jks
searchguard.ssl.transport.keystore_password: ${KEY_PASS}
searchguard.ssl.transport.truststore_filepath: certs/root.jks
searchguard.ssl.transport.truststore_password: ${KEY_PASS}
thread_pool:
  bulk:
    queue_size: 10000

ikakavas · November 15, 2019, 9:07am

Please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

yaminisridaran · November 15, 2019, 4:04pm

@ikakavas, I have fixed the formatting and update the post.

system · December 13, 2019, 4:04pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch stopped working on kubernetes Elasticsearch docker	1	415	April 8, 2021
Remote Cluster Connectivity failure Elasticsearch	5	1025	August 11, 2020
Unable to connect to Elasticsearch at http://localhost:9200 on Kubernetes Elasticsearch	4	5816	August 8, 2017
ElasticSearch 0.92 issue when stop Client Node Elasticsearch	1	331	July 6, 2017
Elasticsearch 3 node cluster failing if master is down second time Elasticsearch	9	3145	June 25, 2017

Elasticsearch cluster is down

Related topics