Need help in understanding Elasticsearch configuration with TLS/SSL

kasim123 · February 11, 2021, 10:51am

Hi Team,

We have a EFK cluster (7.4.x) which was setup more than a year ago with x-pack enabled. At that I followed official document and created self-signed certificates using elastisearch-certutil and that was working with x-pack security enabled. Recently, started to setup new EFK stack ( 7.9.0) using same self-signed certifcates and setup was done, elastic pods are in running state and kibana as well. While working in new setup, I found a warning message stating to use kibana_system user instead of elastic user for kibana. I had following this link Problems enabling authentication to elasticsearch - #3 by tenet_testuser1 and it is being mentioned to use bin/elasticsearch-setup-passwords interactive for setting kibana_system user password. However, when I issued bin/elasticsearch-setup-passwords interactive command, I ended up with an error.

I followed below given link and generated certs.

[root@kubespray certs-with-CN-elasticsearch]# ls
elastic-certificate.pem  elasticsearch-master.p12  elastic-stack-ca.p12  elastic-stack-ca.pem  secret.txt
[root@kubespray certs-with-CN-elasticsearch]#

Here is the error message.

bash-4.4$ bin/elasticsearch-setup-passwords auto
10:30:07.572 [main] WARN  org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [10.233.91.97]; the server provided a certificate with subject name [CN=elasticsearch-master] and fingerprint [38604b086b4fc22b7a507b93da5570cbe65305d2]; the certificate has subject alternative names [DNS:elasticsearch-master]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [70ce3eeeb307dd81297947e2701be35c8d9caa5b] {trusted issuer}) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is trusted in this ssl context ([xpack.security.http.ssl])
java.security.cert.CertificateException: No subject alternative names matching IP address 10.233.91.97 found
        at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165) ~[?:?]
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:101) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:426) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:238) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:78) [elasticsearch-ssl-config-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) [?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) [?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:199) [?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) [?:?]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1488) [?:?]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1394) [?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:441) [?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:412) [?:?]
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) [?:?]
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) [?:?]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142) [?:?]
        at org.elasticsearch.xpack.core.common.socket.SocketAccess.lambda$doPrivileged$0(SocketAccess.java:43) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.security.AccessController.doPrivileged(AccessController.java:554) [?:?]
        at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:42) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.execute(CommandLineHttpClient.java:108) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:302) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:131) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:80) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:109) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

SSL connection to https://10.233.91.97:9200/_security/_authenticate?pretty failed: No subject alternative names matching IP address 10.233.91.97 found
Please check the elasticsearch SSL settings under xpack.security.http.ssl.

ERROR: Failed to establish SSL connection to elasticsearch at https://10.233.91.97:9200/_security/_authenticate?pretty.
bash-4.4$

i did not what the error means. Do I have to create new certificates by using SAN name again. I could see elasticesarch and kibana pods are in RUNNING state. I can access kibana console.

Here is my values.yaml file for elasticsearch.

esConfig:
  elasticsearch.yml: |
     xpack.security.enabled: true
     xpack.security.transport.ssl.enabled: true
     xpack.security.transport.ssl.verification_mode: certificate
     xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12
     xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12
     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12
     xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch-master.p12

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

esJavaOpts: "-Xmx2g -Xms2g"

resources:
  requests:
    cpu: "1000m"
    memory: "2Gi"
  limits:
    cpu: "4000m"
    memory: "4Gi"


volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "rook-ceph-block"
  resources:
    requests:
      storage: 200Gi

protocol: https

I could see cluster health status and indexes using same elastic user credentials.

bash-4.4$ curl -u elastic -k https://localhost:9200/_cluster/health?pretty
Enter host password for user 'elastic':
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 8,
  "active_shards" : 16,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
bash-4.4$

Pls help in understand this issue and how to fix it.

Thanks,

Yang_Wang · February 11, 2021, 11:34am

I think your issue is the same as this one over GitHub CLI tool elasticsearch-setup-passwords does not work if TLS auth is enabled and certificates reference only domain names and not IPs · Issue #68435 · elastic/elasticsearch · GitHub

kasim123 · February 11, 2021, 12:09pm

@Yang_Wang , thanks for the reply.

I resolved the error message by adding following shown entry to elasticsearch.yaml

If hostname verification fails, you can disable this verification by setting xpack.security.http.ssl.verification_mode to certificate

bash-4.4$ bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]

Thanks,

kasim123 · February 11, 2021, 12:14pm

@Yang_Wang ,

I need one more help regarding use kibana_system user instead of elastic user in kibana.yaml file. How do I set those changes kibana.yaml?

One more help, when I execute below given command, i get empty response.

bash-4.4$ curl -u elastic 'http://10.233.49.7:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'elastic':
curl: (52) Empty reply from server
bash-4.4$

Yang_Wang · February 11, 2021, 9:56pm

You can configure the elasticsearch user for Kibana with:

elasticsearch.username=xxx
elasticsearch.password=yyy

in kibana.yml file (documentation)

The Empty reply from server is because the request is sent using http but the server expected https. So please retry with:

curl -u elastic ‘https://10.233.49.7:9200/_security/_authenticate?pretty’

kasim123 · February 12, 2021, 9:58am

@Yang_Wang , I can see the output for curl command

system · March 12, 2021, 9:59am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.