Elasticsearch with xpack.security.enabled throws Cluster is not yet ready

Hello,

I use the following helm chart

Can someone please explain what is pending in order to get elasticsearch running as

> helm install --namespace logging --name elasticsearch -f elasticsearch/values.yaml ./elasticsearch/ --set imageTag=7.3.0 --set replicas=1 --set cluster.xpackEnable=true --debug

with below config:

  elasticsearch.yml: |
    xpack.security.enabled: "true"

I have the following inputs in logs:

Readiness probe failed: Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )

Hello,
I'm guessing you are running a multi node cluster, seems like not all nodes have NOT joined the cluster yet.
Are all nodes at the same version?

post your elasticsearch log file from all your nodes.

Thanks for checking @Preakness

I use 3 similar nodes and deployment is performed via helm chart.

1:

> {"type": "server", "timestamp": "2019-09-05T14:04:06,310+0000", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "Active license is now [BASIC]; Security is enabled"  }
> {"type": "server", "timestamp": "2019-09-05T14:06:01,773+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "node-join[{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 16, reason: added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}"  }
> {"type": "server", "timestamp": "2019-09-05T14:06:03,506+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: Publication{term=2, version=16}"  }

2:

{"type": "server", "timestamp": "2019-09-05T14:04:06,240+0000", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "Active license is now [BASIC]; Security is enabled"  }
{"type": "server", "timestamp": "2019-09-05T14:04:06,246+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "waiting for elected master node [{elasticsearch-master-2}{_mMfhYNSRLGy6l0yTa-oag}{-vYg1HDgT1e63jbp_POWLQ}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)"  }
{"type": "server", "timestamp": "2019-09-05T14:04:07,075+0000", "level": "INFO", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "waiting for elected master node [{elasticsearch-master-2}{_mMfhYNSRLGy6l0yTa-oag}{-vYg1HDgT1e63jbp_POWLQ}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)"  }
{"type": "server", "timestamp": "2019-09-05T14:06:01,805+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "cluster.uuid": "XXXXXMM6QNWeFmFCKtXXXX", "node.id": "6QIoGNbHS5Sg1OtcjvhGtw",  "message": "added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.yyy}{172.30.xxx.yyy:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: ApplyCommitRequest{term=2, version=16, sourceNode={elasticsearch-master-2}{_mMfhYNSRLGy6l0yTa-oag}{-vYg1HDgT1e63jbp_POWLQ}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}"  }

3:

{"type": "server", "timestamp": "2019-09-05T14:04:06,310+0000", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtZXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "Active license is now [BASIC]; Security is enabled"  }
{"type": "server", "timestamp": "2019-09-05T14:06:01,773+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtZXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "node-join[{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 16, reason: added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}"  }
{"type": "server", "timestamp": "2019-09-05T14:06:03,506+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "cluster.uuid": "XXXXXMM6QNWeFmFCKtZXXX", "node.id": "_mMfhYNSRLGy6l0yTa-oag",  "message": "added {{elasticsearch-master-0}{PBA3XLDqTiekoJy7yX_PLA}{0xSDDM8gRFu02iaKJ9WMFA}{172.30.xxx.xxx}{172.30.xxx.xxx:9300}{dim}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 16, reason: Publication{term=2, version=16}"  }

Another thing, when I add xpack.security.transport.ssl xpack modules, I receive the error below:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

Will the cluster start if you set

xpack.security.transport.ssl.enabled:false

If it will then change it back to true and update these settings by putting the exact paths.

xpack.security.transport.ssl.keystore.path:
xpack.security.transport.ssl.truststore.path:

I've experience odd issues when using relative paths.

Didn't help, the same error, I simply would like to enable security/password on kibana :slight_smile: and I understand that xpack should be enabled on elasticsearch side

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.3.0"

Okay if all you want is xpack security enabled make sure you follow these two guides completely.

Elasticsearch config
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-security.html

Kibana config
https://www.elastic.co/guide/en/kibana/current/using-kibana-with-security.html

I changed docker image to docker.elastic.co/elasticsearch/elasticsearch-oss because of this note Elastic Stack: Elasticsearch, Kibana, Beats & Logstash | Elastic and now I receive this error

"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [_xpack]

with

xpack.security.enabled: true

I receive

"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [xpack.security.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",

Please post your entire elasticsearch.yml file.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.