Mutual tls/ssl on elasticsearch

Elastic Security
1

Hi

I have problem that reminds me topic Mutual tls/ssl on elasticsearch

I'm trying to set mutual tls/ssl authentication between ES server and CURL client. TLS communication works, ES server certificate is accepted by CURL client, but ES server always demands basic authentication, i.e. it requires --user switch on CURL command line and ignores certificate that I pass by means of --cert-type and --cert parameters.

The CURL command is:
curl -v --insecure --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/?pretty

As response I get error 401.

If I add switch --user elastic:elastic, it works, but completely ignores --cert-type and --cert parameters:

curl -v --insecure --user elastic:elastic --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/?pretty

Setup that I made:

Server side:

  1. elasticsearch.yml file contains following lines:
    xpack.security.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: certs\server.p12 # Under config directory
    xpack.security.http.ssl.truststore.path: certs\client.p12 # Under config directory
    xpack.security.http.ssl.client_authentication: required

  2. Set keystore and truststore passwords: abc123
    bin\elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
    bin\elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password

Client side:

  1. Create certificates:
    elasticsearch-certutil cert ca --pem --keep-ca-key --in temp\instance.yml --out temp\certs.zip

  2. Unzip certs.zip.

  3. Create P12 file based on content of certs.zip file:
    elasticsearch-certutil cert --ca-cert temp\certs\ca\ca.crt --ca-key temp\certs\ca\ca.key --pass abc123 -out c:\elasticsearch-7.4.0\temp\client.p12

  4. Copy client.p12 file into elasticsearch config\cert directory.

  5. Make copy of client.p12 named server.p12

  6. Import client.p12 and server.p12 certificates on my local machine.

  7. Set password for elastic user: (elastic)
    bin/elasticsearch-setup-passwords interactive

  8. Run query:
    curl -u elastic:elastic -v -k --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/?pretty

As I already wrote, step 8 works only with -u elastic:elastic switch and ignores --cert-type and --cert switches.
Could you kindly suggest what I'm doing wrong?

2

Hi,

Please see https://www.elastic.co/guide/en/elasticsearch/reference/current/pki-realm.html.

Your configuration is incomplete. What you have right now is mTLS between your client and elasticsearch but on the network layer. The application layer ( elasticsearch ) is not configured to authenticate your user via its client certificate.

3

Thank you very much for fast response. It is very impressive. :grinning: May be you can suggest some shortcut, some type of template that demonstrates a correct configuration of such realm?

4

The docs go to enough detail. Please read through them, attempt a configuration and if you have specific questions/problems with it someone in these forums will be able to help

5

I added the following to ES elasticsearch.yml:

xpack:
security:
authc:
token.enabled: true
realms:
pki:
pki1:
order: 1
delegation.enabled: true
truststore:
path: certs\client.p12
password: abc123

Now I got error 403: "action [cluster:monitor/main] is unauthorized for user [instance]"

I created user named instance (it is correct user name taken from truststore, so at least this part works). After that I added roles superuser and monitoring_user to it, but it did not changed ES server response.

elasticsearch-users useradd instance
elasticsearch-users roles instance -a superuser
elasticsearch-users roles instance -a monitoring_user

6

Your file realm instance user, is not the same as your pki realm instance user. What's more, and I copy from the docs:

When you configure realms in elasticsearch.yml , only the realms you specify are used for authentication. If you also want to use the native or file realms, you must include them in the realm chain.

So, your file realm is not even enabled right now.

What you need to instead is follow the doc I shared above and specifically the section 7. Map roles for PKI users. which shows you how to add a role mapping that would assign the neessary roles to a user that authenticates via your PKI realm

7

Yes, you are right. Output of
curl -X GET https://localhost:9200/_xpack/security/_authenticate?pretty -v -k
shows that user "instance" form realm "pki1" has no roles assigned.

8

Thank you very much.
Eventually I succeeded to define user "instance" correctly. Output of:
curl https://localhost:9200/_xpack/security/_authenticate?pretty -v -k
command returns:

{
"username" : "instance",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"pki_dn" : "CN=instance"
},
"enabled" : true,
"authentication_realm" : {
"name" : "pki1",
"type" : "pki"
},
"lookup_realm" : {
"name" : "pki1",
"type" : "pki"
}
}

But :grinning:!

The credential parameters --cert-type and --cert that I pass in query
curl -u elastic:elastic -v -k --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/?pretty
are still ignored by ES.

The output is same as for query
curl -v --insecure https://127.0.0.1:9200/?pretty

How do I know that mutual TLS is took place?

Actually it is not a clever question. If ES server knows that client tries to connect as user "instance", it should be the client that sends certificate of this user to the server.
But how the CURL client knows that it issues request as "instance" user?

9

Apologies, I dont follow your question. Can you add the requests with their respective responses from es as you get them (instead of "the output is the same") and maybe re-word your question?

10

Actually the question is how CURL client knows what certificate it should send to the ES server if I do not specify it on command line.

Two below CURL queries:
With -cert-type and --cert certification parameters:
curl -v -k --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/?pretty

Without certification parameters:
curl -v --insecure https://127.0.0.1:9200/?pretty

receive same response:
{
"name" : "DANIELS-HRLP",
"cluster_name" : "local_ide",
"cluster_uuid" : "hv6ezrLHQI-7o5vpWhsYTA",
"version" : {
"number" : "7.4.0",
"build_flavor" : "default",
"build_type" : "zip",
"build_hash" : "22e1767283e61a198cb4db791ea66e3f11ab9910",
"build_date" : "2019-09-27T08:36:48.569419Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

It seems that command line certification parameters are ignored. If this is a case how CURL client knows what certificate it should use for communication with ES server?

11

It doesn't.

If that's the case you either have security disabled, or anonymous access enabled in your elasticsearch cluster. Can you share your elasticsearch.yml in it's entirety?

What's the response from

curl -v -k --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/_security/_authenticate

and

curl -v --insecure https://127.0.0.1:9200/_security/_authenticate

?

12

1. Output of command:

curl -v -k --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/_security/_authenticate

c:\elasticsearch-7.4.0>curl -v -k --cert-type P12 --cert C:\elasticsearch-7.4.0\config\certs\client.p12:abc123 https://127.0.0.1:9200/_security/_authenticate
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: using IP address, SNI is not supported by OS.
* schannel: sending initial handshake data: sending 147 bytes...
* schannel: sent initial handshake data: sent 147 bytes
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 2203
* schannel: encrypted data buffer: offset 2203 length 4096
* schannel: sending next handshake data: sending 1262 bytes...
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 107
* schannel: encrypted data buffer: offset 107 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 3/3)
* schannel: stored credential handle in session cache
> GET /_security/_authenticate HTTP/1.1
> Host: 127.0.0.1:9200
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 389
* schannel: encrypted data buffer: offset 389 length 103424
* schannel: decrypted data length: 309
* schannel: decrypted data added: 309
* schannel: decrypted data cached: offset 309 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 309 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 309
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 222
<
{"username":"instance","roles":["superuser"],"full_name":null,"email":null,"metadata":{"pki_dn":"CN=instance"},"enabled":true,"authentication_realm":{"name":"pki1","type":"pki"},"lookup_realm":{"name":"pki1","type":"pki"}}* Connection #0 to host 127.0.0.1 left intact

2. Output of command:
curl -v --insecure https://127.0.0.1:9200/_security/_authenticate

c:\elasticsearch-7.4.0>curl -v --insecure https://127.0.0.1:9200/_security/_authenticate
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: using IP address, SNI is not supported by OS.
* schannel: sending initial handshake data: sending 147 bytes...
* schannel: sent initial handshake data: sent 147 bytes
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 2203
* schannel: encrypted data buffer: offset 2203 length 4096
* schannel: sending next handshake data: sending 1262 bytes...
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 107
* schannel: encrypted data buffer: offset 107 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 3/3)
* schannel: stored credential handle in session cache
> GET /_security/_authenticate HTTP/1.1
> Host: 127.0.0.1:9200
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 389
* schannel: encrypted data buffer: offset 389 length 103424
* schannel: decrypted data length: 309
* schannel: decrypted data added: 309
* schannel: decrypted data cached: offset 309 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 309 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 309
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 222
<
{"username":"instance","roles":["superuser"],"full_name":null,"email":null,"metadata":{"pki_dn":"CN=instance"},"enabled":true,"authentication_realm":{"name":"pki1","type":"pki"},"lookup_realm":{"name":"pki1","type":"pki"}}* Connection #0 to host 127.0.0.1 left intact

3. Content of elacticsearch.yml file:


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: local_ide
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
logger.org.elasticsearch.transport: debug

path.repo: C:/elasticsearch-7.4.0/repo

#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 127.0.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true



thread_pool.search.size: 100
thread_pool.search.queue_size: 500
thread_pool.search.min_queue_size: 10
thread_pool.search.max_queue_size: 1000
thread_pool.search.auto_queue_frame_size: 2000
thread_pool.search.target_response_time: 1s
indices.requests.cache.size: 10%

# @@DS
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs\server.p12 # Under config directory
xpack.security.http.ssl.truststore.path: certs\client.p12 # Under config directory
xpack.security.http.ssl.client_authentication: required


xpack:
  security:
    authc:
      token.enabled: true
      realms:
        pki:
          pki1:
            order: 1
            delegation.enabled: true
            truststore:
              path: certs\client.p12
#              password: abc123
# @@DS
13

I suspect this is due to curl's session caching on windows (using schannel)

Can you try adding --no-sessionid to your curl commands and see if that changes the behaviour?

1 Like
14

The result is same. I also paid attention on this message. I tried to run CURL from different instances of commander, I even rebooted my computer. It doesn't make any difference.

curl -v -k --no-sessionid https://127.0.0.1:9200/_security/_authenticate?pretty

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: using IP address, SNI is not supported by OS.
* schannel: sending initial handshake data: sending 147 bytes...
* schannel: sent initial handshake data: sent 147 bytes
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 2203
* schannel: encrypted data buffer: offset 2203 length 4096
* schannel: sending next handshake data: sending 1262 bytes...
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 107
* schannel: encrypted data buffer: offset 107 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 3/3)
> GET /_security/_authenticate?pretty HTTP/1.1
> Host: 127.0.0.1:9200
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 485
* schannel: encrypted data buffer: offset 485 length 103424
* schannel: decrypted data length: 403
* schannel: decrypted data added: 403
* schannel: decrypted data cached: offset 403 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 403 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 403
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 316
<
{
  "username" : "instance",
  "roles" : [
    "superuser"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "pki_dn" : "CN=instance"
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "pki1",
    "type" : "pki"
  },
  "lookup_realm" : {
    "name" : "pki1",
    "type" : "pki"
  }
}
* Connection #0 to host 127.0.0.1 left intact
15

I tried to play with ES configuration and it seems to me that ES always assumes that user should be taken from certificate that is defined by:
xpack.security.http.ssl.truststore.path: certs\client.p12

After that it tries to discover role of this user according to setting of appropriate realm, i.e. the one that points to same certificate (in my case certificate is certs\client.p12, hence selected realm will be pki2):

xpack:
  security:
    authc:
      token.enabled: true
      realms:
        pki:
          pki1:
            order: 1
            delegation.enabled: true
            truststore:
              path: certs1\client1.p12
              password: abc123
          pki2:
            order: 2
            delegation.enabled: true
            truststore:
              path: certs\client.p12
              password: abc123

In this case output of CURL command
curl -v -k --no-sessionid https://127.0.0.1:9200/_security/_authenticate?pretty
shows that user instance from realm pki2 is selected:

c:\elasticsearch-7.4.0>curl -v -k --no-sessionid  https://127.0.0.1:9200/_security/_authenticate?pretty
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: using IP address, SNI is not supported by OS.
* schannel: sending initial handshake data: sending 147 bytes...
* schannel: sent initial handshake data: sent 147 bytes
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 2203
* schannel: encrypted data buffer: offset 2203 length 4096
* schannel: sending next handshake data: sending 1262 bytes...
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 107
* schannel: encrypted data buffer: offset 107 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 3/3)
> GET /_security/_authenticate?pretty HTTP/1.1
> Host: 127.0.0.1:9200
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 485
* schannel: encrypted data buffer: offset 485 length 103424
* schannel: decrypted data length: 403
* schannel: decrypted data added: 403
* schannel: decrypted data cached: offset 403 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 403 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 403
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 316
<
{
  "username" : "instance",
  "roles" : [
    "superuser"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "pki_dn" : "CN=instance"
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "pki2",
    "type" : "pki"
  },
  "lookup_realm" : {
    "name" : "pki2",
    "type" : "pki"
  }
}
* Connection #0 to host 127.0.0.1 left intact

If I change path setting of realm pki2 to point to different certificate, e.g. certs1\client1.p12 which defines another user with same name instance, ES fails to discover a role of this user:

xpack:
  security:
    authc:
      token.enabled: true
      realms:
        pki:
          pki1:
            order: 1
            delegation.enabled: true
            truststore:
              path: certs1\client1.p12
              password: abc123
          pki2:
            order: 2
            delegation.enabled: true
            truststore:
              path: certs1\client1.p12
              password: abc123

In this case output of CURL command
curl -v -k --no-sessionid https://127.0.0.1:9200/_security/_authenticate?pretty
shows that ES is unable to authenticate user instance:

c:\elasticsearch-7.4.0>curl -v -k --no-sessionid  https://127.0.0.1:9200/_security/_authenticate?pretty
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: using IP address, SNI is not supported by OS.
* schannel: sending initial handshake data: sending 147 bytes...
* schannel: sent initial handshake data: sent 147 bytes
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 2203
* schannel: encrypted data buffer: offset 2203 length 4096
* schannel: sending next handshake data: sending 1262 bytes...
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 107
* schannel: encrypted data buffer: offset 107 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 3/3)
> GET /_security/_authenticate?pretty HTTP/1.1
> Host: 127.0.0.1:9200
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 1045
* schannel: encrypted data buffer: offset 1045 length 103424
* schannel: decrypted data length: 966
* schannel: decrypted data added: 966
* schannel: decrypted data cached: offset 966 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 966 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 966
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< content-type: application/json; charset=UTF-8
< content-length: 742
<
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [instance] for REST request [/_security/_authenticate?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [instance] for REST request [/_security/_authenticate?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}
* Connection #0 to host 127.0.0.1 left intact

Any ideas?

16

I think this is still winSSL ( schannel ) sending the client certificate on its own. See Performing Authentication Using Schannel - Win32 apps | Microsoft Learn

When the server requests client authentication, the client must send the server one of its certificates. By default, Schannel will, with no notification to the client, attempt to locate a client certificate and send it to the server

Nope, this is not how the PKI realm works.

"Selected realm" ( or more correctly the realm that managed to authenticate your user ) is pki2 because the certificate that the schannel presents is not trusted by pki1 ( PKI1 trusts only certificates that are either contained in client1.p12 or signed by certificates contained in client1.p12 ) . We try realms in order, so when pki1 fails, elasticsearch attempts to authenticate you via pki2 and that succeeds, as expected.

It doesn't "fail to discover the role of this user", it fails to authenticate the user at all. This happens because now both your PKI realms only trust client1.p12, so again only certificates that are either contained in client1.p12 or signed by certificates contained in client1.p12, and your client still sends the original certificate it has cached.

You can be sure that it is your client that does actually send the certificate and not a bug in elasticsearch by setting -Djavax.net.debug=ssl:handshake in your ES_PATH/config/jvm.properties and then check your elasticearch logs to see the TLS handshake which will contain the Certificate message that your client sends with its client certificate.

As to how to stop curl with winssl/schannel from sending the certificate if not explicitly told so, I'm at a loss.

17

You mean jvm.options file?
I tried to put this switch, but it was ignored. I'm not able to find it among JVM run time arguments in log files.

18

Yes, I mistyped. See Setting JVM options | Elasticsearch Guide [7.9] | Elastic

Which file did you edit, where it was located ? You dont see any TLS handshake messages in your logs ? Btw, which version are you running and did you install it ?

19

I'm running ES "7.4.0". I've installed it by myself.
Eventually I edited bin\elasticsearch.bat file and launched ES from command line. The output indeed contains the Certificate, but I have no expertise to claim if it works as you expected. Can you please check the below output and reply?

Using SSLEngineImpl.
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 for TLSv1.1
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], READ: TLSv1.2 Handshake, length = 142
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1586348129 bytes = { 7, 158, 143, 229, 133, 44, 172, 157, 235, 158, 191, 198, 164, 223, 22, 22, 222, 184, 53, 229, 80, 101, 196, 55, 26, 12, 137, 30 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {unknown curve 29, secp256r1, secp384r1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA256withRSA, SHA384withRSA, SHA1withRSA, SHA256withECDSA, SHA384withECDSA, SHA1withECDSA, SHA1withDSA, SHA512withRSA, SHA512withECDSA
Unsupported extension type_35, data:
Extension extended_master_secret
Extension renegotiation_info, renegotiated_connection: <empty>
***
%% Initialized:  [Session-2, SSL_NULL_WITH_NULL_NULL]
Standard ciphersuite chosen: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
%% Negotiating:  [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
*** ServerHello, TLSv1.2
RandomCookie:  GMT: 1586348129 bytes = { 150, 117, 99, 53, 221, 70, 49, 142, 235, 227, 251, 188, 49, 63, 92, 176, 96, 234, 213, 0, 177, 208, 164, 55, 130, 175, 164, 200 }
Session ID:  {95, 142, 192, 97, 235, 233, 173, 102, 190, 101, 160, 199, 64, 254, 83, 69, 193, 178, 187, 50, 17, 90, 44, 189, 29, 29, 71, 249, 23, 89, 147, 243}
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
Extension extended_master_secret
***
Cipher suite:  TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: CN=instance
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  modulus: 17339886944325530558202443720358344879439047534889386346318357967262331383966095545655428747520682500883334211781901276174982747348954783590757911297900500102938690218140226726992815557172532131863894853270381383637415822868871339187419766415195433420236532818876970462758404356176782008512898162845916076003312372923083663040486401363551406210114674646758744281114098008882890154276800272681906334379527527779199726049722239387213880783566646912837609021273607991252783517932538298227034667284170510889648818375102223472312623211370128045338757452619904598495599783495786718579027307925805255097609764074717329821201
  public exponent: 65537
  Validity: [From: Sun Oct 18 20:07:36 IDT 2020,
               To: Wed Oct 18 20:07:36 IDT 2023]
  Issuer: CN=Elastic Certificate Tool Autogenerated CA
  SerialNumber: [    2f3da8f6 e780d9d8 139983f7 44641e2f 2d5a777b]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BD 77 6E 72 E4 F7 50 B3   59 16 8F F7 4F 21 5A 08  .wnr..P.Y...O!Z.
0010: 62 F1 C2 6D                                        b..m
]
]

[2]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[3]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 34 8F 0E 3A DA FD 6A 1A   02 E6 98 66 12 92 53 30  4..:..j....f..S0
0010: 1C C5 ED CE                                        ....
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 2A 2A 31 F6 54 FD 25 98   FA 75 64 1A EA 77 FC 0B  **1.T.%..ud..w..
0010: 04 36 38 06 DC 9C 69 09   CC 80 AF C4 A3 04 06 B0  .68...i.........
0020: 1D 36 47 DE F4 F8 FB 7E   B0 7E 0E 8D 5A E4 16 87  .6G.........Z...
0030: D0 84 4E AD 20 2A E8 CF   8E 84 2D 6C A0 5A 14 E3  ..N. *....-l.Z..
0040: 4C 92 F2 61 CA A1 79 59   B9 3F 7C F7 03 FE B6 91  L..a..yY.?......
0050: E8 F6 0A 6F EE 7E AE D5   13 B4 D0 AF 67 5C 32 9A  ...o........g\2.
0060: F9 05 75 BE 28 28 7E A9   CE D6 4F A4 90 F9 E4 C7  ..u.((....O.....
0070: 55 05 83 46 E7 7A 3A 1B   F2 0A FE 14 DD 70 D8 47  U..F.z:......p.G
0080: C2 CE DE 62 46 25 16 D0   EB 11 BA 1D 8F 1D EA 6A  ...bF%.........j
0090: 76 0D E4 BA 40 83 9F 3E   1E D2 F4 11 AA F7 68 3D  v...@..>......h=
00A0: F0 0D E1 EA F2 BC 29 5D   97 91 FC 84 D8 10 49 6B  ......)]......Ik
00B0: 71 AB F7 02 D4 C1 6F BB   1F 05 44 32 20 78 A4 11  q.....o...D2 x..
00C0: 83 D9 62 00 25 C8 AB 6D   59 5A 0B C0 15 AB 17 01  ..b.%..mYZ......
00D0: D5 E3 51 03 4E B0 1E A0   94 D6 AD D2 06 8D 28 4A  ..Q.N.........(J
00E0: 04 F4 15 76 D3 8C 87 68   7B CB F4 E5 96 AB 83 1A  ...v...h........
00F0: 34 51 BE 83 65 89 47 74   21 FE D4 F6 84 8B 4B 4D  4Q..e.Gt!.....KM

]
chain [1] = [
[
  Version: V3
  Subject: CN=Elastic Certificate Tool Autogenerated CA
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  modulus: 20994180528997898935323779891755216516753753319468430641704669575194834315119893382939525703587130447092126985286605968792018283876486705923654540568549800734171492095704388087663418756790091283104937796049675050286011599259137947783157116071155104227272769428594439079155744910120936903631953889635362158877700477678081959024315588387754014525050542610228114833885332911769617925234385073443225001776646665911623887362041714617288290718366561508089654408453552564007247855937719551622188549305633478037266503726892034519888772095207592338686657870190919351615116130848550277785499876223423704931086080764689071230007
  public exponent: 65537
  Validity: [From: Sun Oct 18 20:02:16 IDT 2020,
               To: Wed Oct 18 20:02:16 IDT 2023]
  Issuer: CN=Elastic Certificate Tool Autogenerated CA
  SerialNumber: [    b9c11120 55e72f47 c30916f2 be1e452e bd51363d]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BD 77 6E 72 E4 F7 50 B3   59 16 8F F7 4F 21 5A 08  .wnr..P.Y...O!Z.
0010: 62 F1 C2 6D                                        b..m
]
]

[2]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

[3]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: BD 77 6E 72 E4 F7 50 B3   59 16 8F F7 4F 21 5A 08  .wnr..P.Y...O!Z.
0010: 62 F1 C2 6D                                        b..m
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 44 F5 99 A4 DD 55 09 85   9E D6 40 89 C0 40 96 93  D....U....@..@..
0010: C6 6F 66 FD 59 8F 8A 16   2D 89 3D B8 5E EF A5 55  .of.Y...-.=.^..U
0020: DC 51 55 22 C8 3A B8 AF   65 16 79 7C 2D A2 7D DD  .QU".:..e.y.-...
0030: D4 C2 D3 F5 2C 83 13 B0   CC 58 3F FC 65 D5 3A 63  ....,....X?.e.:c
0040: B6 50 AC F2 C6 52 85 95   60 F7 E3 5A B0 DE B5 7D  .P...R..`..Z....
0050: E4 E7 7B E5 B1 11 3A 48   1E 6B 3C F3 43 90 93 BC  ......:H.k<.C...
0060: C1 A4 87 7C E9 FA E6 CA   70 A4 D7 29 49 21 0E 38  ........p..)I!.8
0070: 57 E8 F8 FE 3E 6E 7D 79   0D DF B0 5D 48 03 E1 BD  W...>n.y...]H...
0080: A2 0A 38 C4 4F 39 62 E7   2C DF 8E 60 B0 DB C8 27  ..8.O9b.,..`...'
0090: A7 D3 41 82 13 45 27 F7   95 39 04 C7 97 B7 C3 70  ..A..E'..9.....p
00A0: 55 A7 0A 58 E9 0A 31 D8   46 D2 A2 A6 F1 A7 D3 1C  U..X..1.F.......
00B0: 0E 99 28 45 36 C0 56 0D   11 5E 5F FB 17 2D 2B 91  ..(E6.V..^_..-+.
00C0: BE 59 0A FF 3C C8 D5 BB   97 6E 8C 2F AD 27 BF FD  .Y..<....n./.'..
00D0: C1 68 55 A1 A6 5E 91 3B   DC 2F 18 89 13 E0 0A 88  .hU..^.;./......
00E0: D1 8B 83 89 3F 9B 21 9C   CD FE B3 46 E0 7E DB 42  ....?.!....F...B
00F0: F9 C7 AC 45 FB D8 14 18   62 92 63 B4 34 7B 85 74  ...E....b.c.4..t

]
20 
***
*** ECDH ServerKeyExchange
Signature Algorithm SHA256withRSA
Server key: Sun EC public key, 256 bits
  public x coord: 82871139569738080845858649466631727164383793232454692198701608423533987590015
  public y coord: 79842910319602270094728788433795662245043210926635649630384319282014303073943
  parameters: secp256r1 [NIST P-256, X9.62 prime256v1] (1.2.840.10045.3.1.7)
*** CertificateRequest
Cert Types: RSA, DSS, ECDSA
Supported Signature Algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
Cert Authorities:
<CN=instance>
<CN=Elastic Certificate Tool Autogenerated CA>
*** ServerHelloDone
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], WRITE: TLSv1.2 Handshake, length = 2198
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], READ: TLSv1.2 Handshake, length = 1150
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: CN=instance
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  modulus: 17339886944325530558202443720358344879439047534889386346318357967262331383966095545655428747520682500883334211781901276174982747348954783590757911297900500102938690218140226726992815557172532131863894853270381383637415822868871339187419766415195433420236532818876970462758404356176782008512898162845916076003312372923083663040486401363551406210114674646758744281114098008882890154276800272681906334379527527779199726049722239387213880783566646912837609021273607991252783517932538298227034667284170510889648818375102223472312623211370128045338757452619904598495599783495786718579027307925805255097609764074717329821201
  public exponent: 65537
  Validity: [From: Sun Oct 18 20:07:36 IDT 2020,
               To: Wed Oct 18 20:07:36 IDT 2023]
  Issuer: CN=Elastic Certificate Tool Autogenerated CA
  SerialNumber: [    2f3da8f6 e780d9d8 139983f7 44641e2f 2d5a777b]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BD 77 6E 72 E4 F7 50 B3   59 16 8F F7 4F 21 5A 08  .wnr..P.Y...O!Z.
0010: 62 F1 C2 6D                                        b..m
]
]

[2]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[3]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 34 8F 0E 3A DA FD 6A 1A   02 E6 98 66 12 92 53 30  4..:..j....f..S0
0010: 1C C5 ED CE                                        ....
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 2A 2A 31 F6 54 FD 25 98   FA 75 64 1A EA 77 FC 0B  **1.T.%..ud..w..
0010: 04 36 38 06 DC 9C 69 09   CC 80 AF C4 A3 04 06 B0  .68...i.........
0020: 1D 36 47 DE F4 F8 FB 7E   B0 7E 0E 8D 5A E4 16 87  .6G.........Z...
0030: D0 84 4E AD 20 2A E8 CF   8E 84 2D 6C A0 5A 14 E3  ..N. *....-l.Z..
0040: 4C 92 F2 61 CA A1 79 59   B9 3F 7C F7 03 FE B6 91  L..a..yY.?......
0050: E8 F6 0A 6F EE 7E AE D5   13 B4 D0 AF 67 5C 32 9A  ...o........g\2.
0060: F9 05 75 BE 28 28 7E A9   CE D6 4F A4 90 F9 E4 C7  ..u.((....O.....
0070: 55 05 83 46 E7 7A 3A 1B   F2 0A FE 14 DD 70 D8 47  U..F.z:......p.G
0080: C2 CE DE 62 46 25 16 D0   EB 11 BA 1D 8F 1D EA 6A  ...bF%.........j
0090: 76 0D E4 BA 40 83 9F 3E   1E D2 F4 11 AA F7 68 3D  v...@..>......h=
00A0: F0 0D E1 EA F2 BC 29 5D   97 91 FC 84 D8 10 49 6B  ......)]......Ik
00B0: 71 AB F7 02 D4 C1 6F BB   1F 05 44 32 20 78 A4 11  q.....o...D2 x..
00C0: 83 D9 62 00 25 C8 AB 6D   59 5A 0B C0 15 AB 17 01  ..b.%..mYZ......
00D0: D5 E3 51 03 4E B0 1E A0   94 D6 AD D2 06 8D 28 4A  ..Q.N.........(J
00E0: 04 F4 15 76 D3 8C 87 68   7B CB F4 E5 96 AB 83 1A  ...v...h........
00F0: 34 51 BE 83 65 89 47 74   21 FE D4 F6 84 8B 4B 4D  4Q..e.Gt!.....KM

]
***
Found trusted certificate:
[
[
  Version: V3
  Subject: CN=instance
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  modulus: 17339886944325530558202443720358344879439047534889386346318357967262331383966095545655428747520682500883334211781901276174982747348954783590757911297900500102938690218140226726992815557172532131863894853270381383637415822868871339187419766415195433420236532818876970462758404356176782008512898162845916076003312372923083663040486401363551406210114674646758744281114098008882890154276800272681906334379527527779199726049722239387213880783566646912837609021273607991252783517932538298227034667284170510889648818375102223472312623211370128045338757452619904598495599783495786718579027307925805255097609764074717329821201
  public exponent: 65537
  Validity: [From: Sun Oct 18 20:07:36 IDT 2020,
               To: Wed Oct 18 20:07:36 IDT 2023]
  Issuer: CN=Elastic Certificate Tool Autogenerated CA
  SerialNumber: [    2f3da8f6 e780d9d8 139983f7 44641e2f 2d5a777b]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: BD 77 6E 72 E4 F7 50 B3   59 16 8F F7 4F 21 5A 08  .wnr..P.Y...O!Z.
0010: 62 F1 C2 6D                                        b..m
]
]

[2]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[3]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 34 8F 0E 3A DA FD 6A 1A   02 E6 98 66 12 92 53 30  4..:..j....f..S0
0010: 1C C5 ED CE                                        ....
]
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 2A 2A 31 F6 54 FD 25 98   FA 75 64 1A EA 77 FC 0B  **1.T.%..ud..w..
0010: 04 36 38 06 DC 9C 69 09   CC 80 AF C4 A3 04 06 B0  .68...i.........
0020: 1D 36 47 DE F4 F8 FB 7E   B0 7E 0E 8D 5A E4 16 87  .6G.........Z...
0030: D0 84 4E AD 20 2A E8 CF   8E 84 2D 6C A0 5A 14 E3  ..N. *....-l.Z..
0040: 4C 92 F2 61 CA A1 79 59   B9 3F 7C F7 03 FE B6 91  L..a..yY.?......
0050: E8 F6 0A 6F EE 7E AE D5   13 B4 D0 AF 67 5C 32 9A  ...o........g\2.
0060: F9 05 75 BE 28 28 7E A9   CE D6 4F A4 90 F9 E4 C7  ..u.((....O.....
0070: 55 05 83 46 E7 7A 3A 1B   F2 0A FE 14 DD 70 D8 47  U..F.z:......p.G
0080: C2 CE DE 62 46 25 16 D0   EB 11 BA 1D 8F 1D EA 6A  ...bF%.........j
0090: 76 0D E4 BA 40 83 9F 3E   1E D2 F4 11 AA F7 68 3D  v...@..>......h=
00A0: F0 0D E1 EA F2 BC 29 5D   97 91 FC 84 D8 10 49 6B  ......)]......Ik
00B0: 71 AB F7 02 D4 C1 6F BB   1F 05 44 32 20 78 A4 11  q.....o...D2 x..
00C0: 83 D9 62 00 25 C8 AB 6D   59 5A 0B C0 15 AB 17 01  ..b.%..mYZ......
00D0: D5 E3 51 03 4E B0 1E A0   94 D6 AD D2 06 8D 28 4A  ..Q.N.........(J
00E0: 04 F4 15 76 D3 8C 87 68   7B CB F4 E5 96 AB 83 1A  ...v...h........
00F0: 34 51 BE 83 65 89 47 74   21 FE D4 F6 84 8B 4B 4D  4Q..e.Gt!.....KM

]
*** ECDHClientKeyExchange
ECDH Public value:  { 4, 49, 213, 181, 8, 24, 154, 104, 198, 30, 162, 162, 158, 154, 86, 151, 177, 113, 168, 113, 144, 159, 253, 41, 161, 33, 183, 86, 194, 115, 41, 6, 83, 245, 157, 102, 212, 157, 50, 238, 123, 243, 237, 178, 158, 64, 76, 92, 58, 154, 64, 231, 84, 3, 129, 20, 184, 6, 197, 127, 111, 90, 148, 61, 220 }
SESSION KEYGEN:
PreMaster Secret:
0000: 18 94 52 F7 FE 51 67 BD   6E 85 41 9D 42 70 6B 74  ..R..Qg.n.A.Bpkt
0010: 2B 95 9D 7B AA C7 80 30   64 35 7A 5E 53 B6 0D 0A  +......0d5z^S...
CONNECTION KEYGEN:
Client Nonce:
0000: 5F 8E C0 61 07 9E 8F E5   85 2C AC 9D EB 9E BF C6  _..a.....,......
0010: A4 DF 16 16 DE B8 35 E5   50 65 C4 37 1A 0C 89 1E  ......5.Pe.7....
Server Nonce:
0000: 5F 8E C0 61 96 75 63 35   DD 46 31 8E EB E3 FB BC  _..a.uc5.F1.....
0010: 31 3F 5C B0 60 EA D5 00   B1 D0 A4 37 82 AF A4 C8  1?\.`......7....
Master Secret:
0000: FA 28 F1 9C AE 1E C0 9C   90 0E 8F 06 EF C7 A5 7E  .(..............
0010: 4F 9F 7E 3D 65 E9 22 62   FC 1A E7 EC 10 E5 D2 C4  O..=e."b........
0020: A9 D0 BF 7E 5E C7 D7 CA   71 46 98 6A E7 E3 E1 FD  ....^...qF.j....
Client MAC write Secret:
0000: 10 8C 04 C6 CF B9 F7 11   FC 42 92 24 6B F5 6B 5F  .........B.$k.k_
0010: 69 DE 80 25 EE 53 70 BE   7F AD A4 57 C4 F3 E4 05  i..%.Sp....W....
0020: 2D 3E 36 BF 24 10 CD 5F   A7 B4 03 E5 EF A8 EF 2A  ->6.$.._.......*
Server MAC write Secret:
0000: FE 3C 52 79 2F 4B 9B 30   25 06 AF F2 DC 4C 30 33  .<Ry/K.0%....L03
0010: 36 62 10 C0 4C F6 97 3A   54 8A 1C 10 27 01 65 DE  6b..L..:T...'.e.
0020: 75 C8 3D AD 02 A8 B6 C3   B8 E4 BA 24 FA FE 99 2A  u.=........$...*
Client write key:
0000: 56 AA 86 DD 22 D5 86 F1   A2 1E AA AD E0 52 E2 B4  V..."........R..
0010: A5 A2 79 B4 1C 1D 2F BC   89 E5 E4 59 E1 E1 CC 93  ..y.../....Y....
Server write key:
0000: 48 89 23 FE E9 FA C7 11   6B 8D 2F 25 15 9A 87 3C  H.#.....k./%...<
0010: DE 0A 94 08 30 09 A2 85   94 84 AB 83 F2 00 05 B9  ....0...........
... no IV derived for this protocol
*** CertificateVerify
Signature Algorithm SHA256withRSA
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], READ: TLSv1.2 Change Cipher Spec, length = 1
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], READ: TLSv1.2 Handshake, length = 96
*** Finished
verify_data:  { 80, 205, 17, 115, 159, 68, 78, 68, 44, 121, 233, 9 }
***
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data:  { 229, 245, 8, 139, 199, 7, 76, 102, 139, 17, 92, 16 }
***
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], WRITE: TLSv1.2 Handshake, length = 96
%% Cached server session: [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], WRITE: TLSv1.2 Application Data, length = 403
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], READ: TLSv1.2 Alert, length = 80
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], RECV TLSv1.2 ALERT:  warning, close_notify
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], closeInboundInternal()
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], closeOutboundInternal()
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], SEND TLSv1.2 ALERT:  warning, description = close_notify
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], WRITE: TLSv1.2 Alert, length = 80
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], called closeOutbound()
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], closeOutboundInternal()
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], called closeInbound()
elasticsearch[DANIELS-HRLP][http_server_worker][T#3], closeInboundInternal()