Hi Team,
We have a EFK cluster (7.4.x) which was setup more than a year ago with x-pack enabled. At that I followed official document and created self-signed certificates using elastisearch-certutil
and that was working with x-pack security enabled. Recently, started to setup new EFK stack ( 7.9.0) using same self-signed certifcates and setup was done, elastic pods are in running state and kibana as well. While working in new setup, I found a warning message stating to use kibana_system
user instead of elastic
user for kibana. I had following this link Problems enabling authentication to elasticsearch - #3 by tenet_testuser1 and it is being mentioned to use bin/elasticsearch-setup-passwords interactive
for setting kibana_system
user password. However, when I issued bin/elasticsearch-setup-passwords interactive
command, I ended up with an error.
I followed below given link and generated certs.
[root@kubespray certs-with-CN-elasticsearch]# ls
elastic-certificate.pem elasticsearch-master.p12 elastic-stack-ca.p12 elastic-stack-ca.pem secret.txt
[root@kubespray certs-with-CN-elasticsearch]#
Here is the error message.
bash-4.4$ bin/elasticsearch-setup-passwords auto
10:30:07.572 [main] WARN org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [10.233.91.97]; the server provided a certificate with subject name [CN=elasticsearch-master] and fingerprint [38604b086b4fc22b7a507b93da5570cbe65305d2]; the certificate has subject alternative names [DNS:elasticsearch-master]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [70ce3eeeb307dd81297947e2701be35c8d9caa5b] {trusted issuer}) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is trusted in this ssl context ([xpack.security.http.ssl])
java.security.cert.CertificateException: No subject alternative names matching IP address 10.233.91.97 found
at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165) ~[?:?]
at sun.security.util.HostnameChecker.match(HostnameChecker.java:101) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:426) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:238) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:78) [elasticsearch-ssl-config-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341) [?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) [?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) [?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) [?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:199) [?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) [?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1488) [?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1394) [?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:441) [?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:412) [?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) [?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) [?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142) [?:?]
at org.elasticsearch.xpack.core.common.socket.SocketAccess.lambda$doPrivileged$0(SocketAccess.java:43) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(AccessController.java:554) [?:?]
at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:42) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.execute(CommandLineHttpClient.java:108) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:302) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:131) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:80) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:109) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
SSL connection to https://10.233.91.97:9200/_security/_authenticate?pretty failed: No subject alternative names matching IP address 10.233.91.97 found
Please check the elasticsearch SSL settings under xpack.security.http.ssl.
ERROR: Failed to establish SSL connection to elasticsearch at https://10.233.91.97:9200/_security/_authenticate?pretty.
bash-4.4$
i did not what the error means. Do I have to create new certificates by using SAN name again. I could see elasticesarch and kibana pods are in RUNNING state. I can access kibana console.
Here is my values.yaml file for elasticsearch.
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
esJavaOpts: "-Xmx2g -Xms2g"
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "4000m"
memory: "4Gi"
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "rook-ceph-block"
resources:
requests:
storage: 200Gi
protocol: https
I could see cluster health status and indexes using same elastic
user credentials.
bash-4.4$ curl -u elastic -k https://localhost:9200/_cluster/health?pretty
Enter host password for user 'elastic':
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 8,
"active_shards" : 16,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
bash-4.4$
Pls help in understand this issue and how to fix it.
Thanks,