Master node failed to retrieve password

Elastic Stack Elasticsearch
1

I have use helm to deploy a cluster with 3 master node + 4 data node.
Master node part of config:

esConfig:
    elasticsearch.yml: |
      bootstrap.memory_lock: true
      xpack.security.enabled: true
      xpack.ml.enabled: false
      path.data: /usr/share/elasticsearch/data0,/usr/share/elasticsearch/data1
      xpack.security.transport.ssl.enabled: true
      xpack.security.transport.ssl.verification_mode: certificate
      xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
      xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds:  
clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"

I have updated [indices.breaker.total.limit] from [100%] to [80%]
Then the data node got this error, and 3 data node become unready cause the readinessProbe check failed.

{"type": "server", "timestamp": "2021-08-03T11:01:31,671Z", "level": "WARN", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "insight", "node.name": "insight-master-0", "message": "unexpected error while indexing monitoring document", "cluster.uuid": "wzBzIPDTQ4KW7IGMMxDuug", "node.id": "q_3QuVnHTbmC6mbIvoX9rQ" ,
"stacktrace": ["org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[insight-data-0][172.100.220.215:9300][indices:data/write/bulk[s]]]; nested: CircuitBreakingException[[parent] Data too large, data for [indices:data/write/bulk[s]] would be [24333526730/22.6gb], which is larger than the limit of [23300197580/21.6gb], real usage: [24333524648/22.6gb], new bytes reserved: [2082/2kb], usages [request=0/0b, fielddata=17444/17kb, in_flight_requests=2082/2kb, model_inference=0/0b, accounting=63851616/60.8mb]];",

For helm, there is a period check to curl http://user:pwd@127.0.0.1:9200, but this is failed, get error below:

{"type": "server", "timestamp": "2021-08-03T11:19:15,212Z", "level": "ERROR", "component": "o.e.x.s.a.e.ReservedRealm", "cluster.name": "insight", "node.name": "insight-master-0", "message": "failed to retrieve password hash for reserved user [elastic]", "cluster.uuid": "wzBzIPDTQ4KW7IGMMxDuug", "node.id": "q_3QuVnHTbmC6mbIvoX9rQ","stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [insight-data-0][172.100.220.215:9300][indices:data/read/get[s]]"

Question:
#1 What data do master node store, will it store the security data, like password for elastic?
#2 Why get this error, it will connect to data node to check the authority?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.