Elasticsearch cluster is down

yaminisridaran · November 14, 2019, 2:59pm

We have deployed elasticsearch cluster on kubernetes cluster.
We have deployed 3 master nodes, 3 client nodes and 5 data nodes.
We are able to ping the endpoint ips mentioned in the logs from the master,client and data nodes, but the logs are showing connection issue (manual ping works inside the containers).

Master node logs:
[2019-11-14T12:40:57,348][WARN ][o.e.c.NodeConnectionsService] [master-fed-elastic-0] failed to connect to node

{client-fed-elastic-1}{X3EsRyYLScSEaumLVWGlDQ}{sXzX4oU_TzCaJPRnkUEnTg}{192.168.186.209}{192.168.186.209:9300} (tried [1] times)
Client node logs:
2019-11-14T12:36:00,265][WARN ][o.e.c.NodeConnectionsService] [client-fed-elastic-0] failed to connect to node {client-fed-elastic-1} {X3EsRyYLScSEaumLVWGlDQ} {9uHMXe8lTrqsICoT4HyIPQ} {192.168.186.209} {192.168.186.209:9300}

(tried [1] times)
[2019-11-14T12:37:28,771][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [client-fed-elastic-0] SSL Problem Received close_notify during handshake
javax.net.ssl.SSLException: Received close_notify during handshake

Data node logs:
[2019-11-14T12:40:45,109][WARN ][c.f.s.s.t.SearchGuardSSLNettyTransport] [data-fed-elastic-0] send message failed [channel: NettyTcpChannel

{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/192.168.124.43:42068}

]

Can you please help us?

A_B · November 14, 2019, 4:02pm

Hi @yaminisridaran,

could you share your Elasticsearch config? Please share as Preformated text, then it is much easier to read

yaminisridaran · November 15, 2019, 8:49am

My Elasticsearch cluster config:

Master node config:

bootstrap:
  memory_lock: false
cluster:
  name: ${CLUSTER_NAME}
discovery:
  zen:
    minimum_master_nodes: ${NUMBER_OF_MASTERS}
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
http:
  compression: true
  cors:
    allow-credentials: true
    allow-headers: X-Requested-With, Content-Type, Content-Length, Authorization
    allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
    enabled: ${HTTP_CORS_ENABLE}
  enabled: ${HTTP_ENABLE}
network.host: 0.0.0.0
node:
  data: ${NODE_DATA}
  ingest: ${NODE_INGEST}
  master: ${NODE_MASTER}
  name: ${NODE_NAME}
path:
  data: /data/data
  logs: /data/log
searchguard.authcz.admin_dn:
- CN=sgadmin,O=Elasticsearch Operator
searchguard.disabled: ${SEARCHGUARD_DISABLED:false}
searchguard.enterprise_modules_enabled: false
searchguard.restapi.roles_enabled:
- sg_all_access
- sg_kibana_user
searchguard.ssl.http.enabled: ${SSL_ENABLE}
searchguard.ssl.http.keystore_filepath: certs/client.jks
searchguard.ssl.http.keystore_password: ${KEY_PASS}
searchguard.ssl.http.truststore_filepath: certs/root.jks
searchguard.ssl.http.truststore_password: ${KEY_PASS}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: certs/node.jks
searchguard.ssl.transport.keystore_password: ${KEY_PASS}
searchguard.ssl.transport.truststore_filepath: certs/root.jks
searchguard.ssl.transport.truststore_password: ${KEY_PASS}
thread_pool:
  bulk:
    queue_size: 10000

Client node config:

bootstrap:
  memory_lock: false
cluster:
  name: ${CLUSTER_NAME}
discovery:
  zen:
    minimum_master_nodes: ${NUMBER_OF_MASTERS}
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
http:
  compression: true
  cors:
    allow-credentials: true
    allow-headers: X-Requested-With, Content-Type, Content-Length, Authorization
    allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
    enabled: ${HTTP_CORS_ENABLE}
  enabled: ${HTTP_ENABLE}
network.host: 0.0.0.0
node:
  data: ${NODE_DATA}
  ingest: ${NODE_INGEST}
  master: ${NODE_MASTER}
  name: ${NODE_NAME}
path:
  data: /data/data
  logs: /data/log
searchguard.authcz.admin_dn:
- CN=sgadmin,O=Elasticsearch Operator
searchguard.disabled: ${SEARCHGUARD_DISABLED:false}
searchguard.enterprise_modules_enabled: false
searchguard.restapi.roles_enabled:
- sg_all_access
- sg_kibana_user
searchguard.ssl.http.enabled: ${SSL_ENABLE}
searchguard.ssl.http.keystore_filepath: certs/client.jks
searchguard.ssl.http.keystore_password: ${KEY_PASS}
searchguard.ssl.http.truststore_filepath: certs/root.jks
searchguard.ssl.http.truststore_password: ${KEY_PASS}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: certs/node.jks
searchguard.ssl.transport.keystore_password: ${KEY_PASS}
searchguard.ssl.transport.truststore_filepath: certs/root.jks
searchguard.ssl.transport.truststore_password: ${KEY_PASS}
thread_pool:
  bulk:
    queue_size: 10000

Data node config:

bootstrap:
  memory_lock: false
cluster:
  name: ${CLUSTER_NAME}
discovery:
  zen:
    minimum_master_nodes: ${NUMBER_OF_MASTERS}
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
http:
  compression: true
  cors:
    allow-credentials: true
    allow-headers: X-Requested-With, Content-Type, Content-Length, Authorization
    allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
    enabled: ${HTTP_CORS_ENABLE}
  enabled: ${HTTP_ENABLE}
network.host: 0.0.0.0
node:
  data: ${NODE_DATA}
  ingest: ${NODE_INGEST}
  master: ${NODE_MASTER}
  name: ${NODE_NAME}
path:
  data: /data/data
  logs: /data/log
searchguard.authcz.admin_dn:
- CN=sgadmin,O=Elasticsearch Operator
searchguard.disabled: ${SEARCHGUARD_DISABLED:false}
searchguard.enterprise_modules_enabled: false
searchguard.restapi.roles_enabled:
- sg_all_access
- sg_kibana_user
searchguard.ssl.http.enabled: ${SSL_ENABLE}
searchguard.ssl.http.keystore_filepath: certs/client.jks
searchguard.ssl.http.keystore_password: ${KEY_PASS}
searchguard.ssl.http.truststore_filepath: certs/root.jks
searchguard.ssl.http.truststore_password: ${KEY_PASS}
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_filepath: certs/node.jks
searchguard.ssl.transport.keystore_password: ${KEY_PASS}
searchguard.ssl.transport.truststore_filepath: certs/root.jks
searchguard.ssl.transport.truststore_password: ${KEY_PASS}
thread_pool:
  bulk:
    queue_size: 10000

ikakavas · November 15, 2019, 9:07am

Please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

yaminisridaran · November 15, 2019, 4:04pm

@ikakavas, I have fixed the formatting and update the post.

system · December 13, 2019, 4:04pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.