Elasticsearch with xpack.security.enabled throws Cluster is not yet ready

O_K · September 5, 2019, 1:17pm

Hello,

I use the following helm chart

Can someone please explain what is pending in order to get elasticsearch running as

> helm install --namespace logging --name elasticsearch -f elasticsearch/values.yaml ./elasticsearch/ --set imageTag=7.3.0 --set replicas=1 --set cluster.xpackEnable=true --debug

with below config:

  elasticsearch.yml: |
    xpack.security.enabled: "true"

I have the following inputs in logs:

Readiness probe failed: Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )

Preakness · September 5, 2019, 1:56pm

Hello,
I'm guessing you are running a multi node cluster, seems like not all nodes have NOT joined the cluster yet.
Are all nodes at the same version?

post your elasticsearch log file from all your nodes.

O_K · September 5, 2019, 2:19pm

Thanks for checking @Preakness

I use 3 similar nodes and deployment is performed via helm chart.

1:

> {"type": "server", "timestamp": "2019-09-05T14:04:06,310+0000", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "Active license is now [BASIC]; Security is enabled"  }
> {"type": "server", "timestamp": "2019-09-05T14:06:01,773+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "node-join[{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 16, reason: added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}"  }
> {"type": "server", "timestamp": "2019-09-05T14:06:03,506+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: Publication{term=2, version=16}"  }

2:

{"type": "server", "timestamp": "2019-09-05T14:04:06,240+0000", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "Active license is now [BASIC]; Security is enabled"  }
{"type": "server", "timestamp": "2019-09-05T14:04:06,246+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "waiting for elected master node [{elasticsearch-master-2}{_mMfhYNSRLGy6l0yTa-oag}{-vYg1HDgT1e63jbp_POWLQ}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)"  }
{"type": "server", "timestamp": "2019-09-05T14:04:07,075+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "waiting for elected master node [{elasticsearch-master-2}{_mMfhYNSRLGy6l0yTa-oag}{-vYg1HDgT1e63jbp_POWLQ}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)"  }
{"type": "server", "timestamp": "2019-09-05T14:06:01,805+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.yyy}{172.30.xxx.yyy:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: ApplyCommitRequest{term=2, version=16, sourceNode={elasticsearch-master-2}{_mMfhYNSRLGy6l0yTa-oag}{-vYg1HDgT1e63jbp_POWLQ}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}"  }

3:

{"type": "server", "timestamp": "2019-09-05T14:04:06,310+0000", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtZXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "Active license is now [BASIC]; Security is enabled"  }
{"type": "server", "timestamp": "2019-09-05T14:06:01,773+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtZXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "node-join[{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 16, reason: added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}"  }
{"type": "server", "timestamp": "2019-09-05T14:06:03,506+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtZXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: Publication{term=2, version=16}"  }

Another thing, when I add xpack.security.transport.ssl xpack modules, I receive the error below:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

Preakness · September 5, 2019, 2:25pm

Will the cluster start if you set

xpack.security.transport.ssl.enabled:false

If it will then change it back to true and update these settings by putting the exact paths.

xpack.security.transport.ssl.keystore.path:
xpack.security.transport.ssl.truststore.path:

I've experience odd issues when using relative paths.

O_K · September 5, 2019, 2:38pm

Didn't help, the same error, I simply would like to enable security/password on kibana and I understand that xpack should be enabled on elasticsearch side

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.3.0"

Preakness · September 5, 2019, 2:41pm

Okay if all you want is xpack security enabled make sure you follow these two guides completely.

Elasticsearch config
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-security.html

Kibana config
https://www.elastic.co/guide/en/kibana/current/using-kibana-with-security.html

O_K · September 5, 2019, 3:22pm

I changed docker image to docker.elastic.co/elasticsearch/elasticsearch-oss because of this note Elastic Stack: Elasticsearch, Kibana, Beats & Logstash | Elastic and now I receive this error

"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [_xpack]

with

xpack.security.enabled: true

I receive

"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",

Preakness · September 5, 2019, 6:37pm

Please post your entire elasticsearch.yml file.

system · October 3, 2019, 6:37pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Enable xpack on a running Elastic cluster Elasticsearch elastic-stack-security	10	2143	October 25, 2021
Kubernetes pods are not ready in a Cluster with xpack.security.enabled: true after the pod restart Elasticsearch	4	4152	October 14, 2020
clusterHealthCheckParams failing in readiness probe on startup Elasticsearch	1	452	July 9, 2020
Kubernetes pods are not ready in a Cluster with xpack.security.enabled: true after the pod restart Elasticsearch docker	1	421	September 20, 2022
After enabling xpack security readinessprobe fails Elasticsearch docker	5	126	July 18, 2024

Elasticsearch with xpack.security.enabled throws Cluster is not yet ready

Related Topics