[x-pack] enabled: Unable to communicate to Elasticsearch using only cacert from client machines

vijayakrishna.rg · August 7, 2019, 3:55pm

Hi,

i have upgraded my cluster from 6.7.0 to 6.8.2 to use x-pack opensource version, after enabling x-pack am able to communicate to elasticsearch using "cacert" and username, password, but i am trying to authenticate only using "cacert" without user and password.

i have added pki realm still am getting below error.

if i use only cacert to authenticate.

curl -XGET  --cacert /etc/elasticsearch/ssl/cacert https://nprod-elastic:443/_cluster/health?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

same command along with username:password

curl -XGET  --cacert /etc/elasticsearch/ssl/cacert -u elastic:changeme https://nprod- elastic:443/_cluster/health?pretty
  {
  "cluster_name" : "escluster-nprod",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
 "number_of_data_nodes" : 3,
  "active_primary_shards" : 2,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

here is my elasticsearch.yml

cluster.name: escluster-nprod
network.host: 0.0.0.0
transport.tcp.port: 9300
http.port: 9200
network.bind_host: 0.0.0.0
discovery.zen.ping.unicast.hosts: elasticnprod-1,elasticnprod-2,elasticnprod-3
node.name: elastic1
#cluster.initial_master_nodes: elastic1
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
cluster.routing.allocation.disk.threshold_enabled: false
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /etc/elasticsearch/ssl/nprod-elastic-client.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/ssl/nprod-elastic-client.pem
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: /etc/elasticsearch/ssl/nprod-elastic-server.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/ssl/nprod-elastic-server.pem
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]
xpack.security.authc.token.enabled: false
xpack.security.http.ssl.client_authentication: optional

xpack:
  security:
    authc:
      realms:
        pki1:
          type: pki
          certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]

Please someone help me with it, Thanks.

ikakavas · August 8, 2019, 4:58am

The --ca-cert parameter is not used for authentication or as credentials of some sort. It just tells your client ( curl in this instance ) which CA certificates to trust when validating the certificate that the server(elasticsearch) is presenting for TLS.

In order to authenticate, your client needs to present a client certificate that is signed by the CA that you have configured elasticsearch to trust for the http layer ( xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]. In curl you pass these parameters in your request with --cert and --key.

See also our PKI realm docs

vijayakrishna.rg · August 8, 2019, 3:22pm

hi Ioannis, Thanks for your response, i have passed key and pem files to curl command still having same issue.

curl -XGET https://nprod-elastic:443/_cluster/health?pretty --key /etc/elasticsearch/ssl/nprod-elastic-client.key --cert /etc/elasticsearch/ssl/nprod-elastic-client.pem
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.


curl -XGET https://nprod-elastic:443/_cluster/health?pretty --key /etc/elasticsearch/ssl/nprod-elastic-client.key --cert /etc/elasticsearch/ssl/nprod-elastic-client.pem --cacert /etc/elasticsearch/ssl/cacert

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "ApiKey",
           "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication token for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
   }
  },
  "status" : 401
}

ikakavas · August 8, 2019, 3:40pm

Please take the time to correctly format your posts, it really enhances the chances that someone will read through this and attempt to help out. You can use the panel on the right to verify that your post looks fine.

You need to pass all three of --cacert, --key and --cert. The --cacert tells curl how to validate it trusts the Elasticsearch's server certificate and --key and --cert provide the key and client certificate that curl would present to Elasticsearch to perform client TLS authentication.

Are you certain that nprod-elastic-client.key is the key that corresponds to the nprod-elastic.pem certificate ? I see you use nprod-elastic-client.key and nprod-elastic-client.pem as the server certificate so it looks like one of the two configs ( or both ) are wrong.

Also is /etc/elasticsearch/ssl/cacert a PEM file or a keystore ? Your use as a value for *.ssl.certificate_authorities implies it's a PEM file but the name implies it's a keystore.

vijayakrishna.rg · August 8, 2019, 4:26pm

Sorry it was a typo, i have corrected in above post.

i have total 5 certs for my elasticsearch.

these are used for http communication
1. nprod-elastic-client.key
2. nprod-elastic-client.pem
3. cacert

these are used for transport communication
1. nprod-elastic-server.key
2. nprod-elastic-server.pem
3. cacert (it's same as above)

cacert is pem formatted file, it's not a keystore.

i was using all three certs --cacert , --key and --cert . for curl command.

ikakavas · August 8, 2019, 4:28pm

As I said above, you need a different pair for the server and a different for the client. You can't use one key/certificate for both sides.

xpack.security.http.ssl.key
xpack.security.http.ssl.certificate

is one thing.

The ones you will use with --key and --certificate is another thing.

vijayakrishna.rg · August 8, 2019, 4:32pm

i have different pair for client and server certs, only common cert is cacert thats my enterprise one, (we use same cacert for all other services in our organization), thats why i have mentioned total 5 certs in above post.

i have also added below 2 parameters into elasticsearch.yml and restarted service.

xpack.security.authc.token.enabled: false
xpack.security.authc.api_key.enabled: false

ikakavas · August 8, 2019, 4:46pm

I'm not sure I follow. You say

i have different pair for client and server certs,

You don't show us that. You only mention 1 pair

1. nprod-elastic-client.key
2. nprod-elastic-client.pem

I repeat that you need different certifiicate/key pairs for the server ( this is elasticsearch) and the client(this is curl) on the http layer.

These do not count neither as server, nor client, these are for the transport layer, which is another thing. When you run curl as a client, you cannot specify the same key and certificate as the ones you use

xpack.security.http.ssl.key
xpack.security.http.ssl.certificate

Please refrain from adding unrelated settings in the middle of troubleshooting. These have nothing to do with your issue, adding extra stuff to your configuration before you solve existing problems will only make it more difficult to resolve your issue.

Slightly unrelated to the exact issue you are describing but this is not a good practice. You use

xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]

which means that you don't validate the certificates but even if you did, setting certificate_authorities to a CA that signs everything in your organization, means that anyone in your organization can get a client certificate from that CA and spin up an Elasticsearch node that can join your cluster.

Please share the exact command you run with curl, the exact and complete , current configuration you use in Elasticsearch , the error you get from curl and the logs from elasticsearch.

vijayakrishna.rg · August 8, 2019, 4:54pm

As you said am taking out extra parameters added recently, i thought these can help to disable token authentication, because below error says missing authentication token
xpack.security.authc.token.enabled: false xpack.security.authc.api_key.enabled: false

Here is the command am running on my elastic cluster:

curl -XGET 'https://nprod-elastic:443/_cluster/health?pretty=true' --key /etc/elasticsearch/ssl/nprod-elastic-client.key --cert /etc/elasticsearch/ssl/nprod-elastic-client.pem --cacert /etc/elasticsearch/ssl/cacert
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication token for REST request [/_cluster/health?pretty=true]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication token for REST request [/_cluster/health?pretty=true]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

ikakavas · August 8, 2019, 4:57pm

Please understand that you need to take the time to review the previous posts , and at least provide the requested information in full. If you don't do that, it's highly unlikely people in these forums will be able/interested to assist you solve your issues. I repeat from my previous post:

vijayakrishna.rg · August 8, 2019, 5:00pm

since it has limitation of number of lines per post, i have divided into 2 posts and took some to format the logs(sorry for delay), i hope you got all the information.

[2019-08-08T16:56:54,377][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-logstash]
[2019-08-08T16:56:54,377][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-ml]
[2019-08-08T16:56:54,377][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-monitoring]
[2019-08-08T16:56:54,378][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-rollup]
[2019-08-08T16:56:54,378][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-security]
[2019-08-08T16:56:54,378][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-sql]
[2019-08-08T16:56:54,378][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-upgrade]
[2019-08-08T16:56:54,378][INFO ][o.e.p.PluginsService     ] [elastic1] loaded module [x-pack-watcher]
[2019-08-08T16:56:54,378][INFO ][o.e.p.PluginsService     ] [elastic1] loaded plugin [repository-azure]
[2019-08-08T16:56:58,324][INFO ][o.e.x.s.a.s.FileRolesStore] [elastic1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-08-08T16:56:59,156][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic1] [controller/60034] [Main.cc@109] controller (64 bit): Version 6.8.2 (Build f4a9b28c0d5114) Copyright (c) 2019 Elasticsearch BV
[2019-08-08T16:56:59,688][DEBUG][o.e.a.ActionModule       ] [elastic1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-08-08T16:56:59,996][INFO ][o.e.d.DiscoveryModule    ] [elastic1] using discovery type [zen] and host providers [settings]
[2019-08-08T16:57:00,948][INFO ][o.e.n.Node               ] [elastic1] initialized
[2019-08-08T16:57:00,948][INFO ][o.e.n.Node               ] [elastic1] starting ...
[2019-08-08T16:57:01,103][INFO ][o.e.t.TransportService   ] [elastic1] publish_address {10.10.10.10:9300}, bound_addresses {[::]:9300}
[2019-08-08T16:57:01,125][INFO ][o.e.b.BootstrapChecks    ] [elastic1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-08-08T16:57:04,292][INFO ][o.e.m.j.JvmGcMonitorService] [elastic1] [gc][3] overhead, spent [380ms] collecting in the last [1.3s]
[2019-08-08T16:57:04,374][INFO ][o.e.c.s.MasterService    ] [elastic1] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {elastic1}{T8rUwmUPQ7ahjd-0D99oxg}{eDcHTIXtQ82xfmNyWF8gXw}{10.10.10.10}{10.10.10.10:9300}{ml.machine_memory=33718145024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-08-08T16:57:04,386][INFO ][o.e.c.s.ClusterApplierService] [elastic1] new_master {elastic1}{T8rUwmUPQ7ahjd-0D99oxg}{eDcHTIXtQ82xfmNyWF8gXw}{10.10.10.10}{10.10.10.10:9300}{ml.machine_memory=33718145024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {elastic1}{T8rUwmUPQ7ahjd-0D99oxg}{eDcHTIXtQ82xfmNyWF8gXw}{10.10.10.10}{10.10.10.10:9300}{ml.machine_memory=33718145024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-08-08T16:57:04,478][INFO ][o.e.h.n.Netty4HttpServerTransport] [elastic1] publish_address {10.10.10.10:9200}, bound_addresses {[::]:9200}
[2019-08-08T16:57:04,479][INFO ][o.e.n.Node               ] [elastic1] started
[2019-08-08T16:57:04,483][INFO ][o.e.c.s.MasterService    ] [elastic1] zen-disco-node-join[{elastic2}{eicr9WWFS6aew6z9B2vlmg}{KKoKyxG3RNaehWc3O83EVA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason: added {{elastic2}{eicr9WWFS6aew6z9B2vlmg}{KKoKyxG3RNaehWc3O83EVA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}
[2019-08-08T16:57:04,521][INFO ][o.e.c.s.ClusterApplierService] [elastic1] added {{elastic2}{eicr9WWFS6aew6z9B2vlmg}{KKoKyxG3RNaehWc3O83EVA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {elastic1}{T8rUwmUPQ7ahjd-0D99oxg}{eDcHTIXtQ82xfmNyWF8gXw}{10.10.10.10}{10.10.10.10:9300}{ml.machine_memory=33718145024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [2] source [zen-disco-node-join[{elastic2}{eicr9WWFS6aew6z9B2vlmg}{KKoKyxG3RNaehWc3O83EVA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]]])
[2019-08-08T16:57:04,526][WARN ][o.e.d.z.ElectMasterService] [elastic1] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [1], total number of master-eligible nodes used for publishing in this round: [2])

[2019-08-08T16:57:05,888][INFO ][o.e.l.LicenseService     ] [elastic1] license [772e0afc-cd99-44d3-a5ff-1a1c529d9899] mode [basic] - valid
[2019-08-08T16:57:05,898][INFO ][o.e.g.GatewayService     ] [elastic1] recovered [1] indices into cluster_state
[2019-08-08T16:57:06,752][INFO ][o.e.c.s.MasterService    ] [elastic1] zen-disco-node-join[{elastic3}{ieNna_CWQNqmHtreQe8caA}{hopwUqyGQwSoVjAmYUaqtA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason: added {{elastic3}{ieNna_CWQNqmHtreQe8caA}{hopwUqyGQwSoVjAmYUaqtA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}
[2019-08-08T16:57:07,546][INFO ][o.e.c.s.ClusterApplierService] [elastic1] added {{elastic3}{ieNna_CWQNqmHtreQe8caA}{hopwUqyGQwSoVjAmYUaqtA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {elastic1}{T8rUwmUPQ7ahjd-0D99oxg}{eDcHTIXtQ82xfmNyWF8gXw}{10.10.10.10}{10.10.10.10:9300}{ml.machine_memory=33718145024, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [6] source [zen-disco-node-join[{elastic3}{ieNna_CWQNqmHtreQe8caA}{hopwUqyGQwSoVjAmYUaqtA}{10.10.10.11}{10.10.10.11:9300}{ml.machine_memory=33718145024, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]]])
[2019-08-08T16:57:07,560][INFO ][o.e.c.r.a.AllocationService] [elastic1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-6-2019.08.08][0]] ...]).
[2019-08-08T16:57:08,305][INFO ][o.e.c.r.a.AllocationService] [elastic1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2019.08.08][0]] ...]).`

vijayakrishna.rg · August 8, 2019, 5:04pm

here is my elasticsearch.yml

cluster.name: escluster-nprod
network.host: 0.0.0.0
transport.tcp.port: 9300
http.port: 9200
network.bind_host: 0.0.0.0
discovery.zen.ping.unicast.hosts: elasticnprod-1,elasticnprod-2,elasticnprod-3
node.name: elastic1
#cluster.initial_master_nodes: elastic1
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
cluster.routing.allocation.disk.threshold_enabled: false
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /etc/elasticsearch/ssl/nprod-elastic-client.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/ssl/nprod-elastic-client.pem
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.key: /etc/elasticsearch/ssl/nprod-elastic-server.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/ssl/nprod-elastic-server.pem
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]


#xpack.security.authc.token.enabled: false
#xpack.security.http.ssl.client_authentication: optional

xpack:
 security:
   authc:
     realms:
       pki1:
         type: pki
         certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]`

ikakavas · August 8, 2019, 5:12pm

vijayakrishna.rg · August 8, 2019, 8:26pm

I am really confused with your comments you need a different pair for the server and a different for the client

i am already having different pair of client and server certs as i mentioned in above postings.

i have client.key and client.pem these are used for http layer and same certs used to connect to elastic cluster from client machines.

1. nprod-elastic-client.key
2. nprod-elastic-client.pem

These server certs are another set of certs to use only for internal node to node communication, these cannot be use to connect from client servers.

1. nprod-elastic-server.key
2. nprod-elastic-server.pem

i really don't know where am using same key and certificate here.

ikakavas · August 8, 2019, 9:38pm

Hi,

No, you don't Let's leave the transport layer ( node to node ) communication out of our discussion. This works and you don't have any issue with it.

Now, the http layer communication:

Elasticsearch is the http server and curl is the http client. Elasticsearch is configured to use TLS for its http layer, and you want the client to also authenticate itself to the server using a TLS Certificate. What I am saying is that you should not use the same certificate both for the http client ( curl ) and the http server ( elasticsearch ) .

You currently use nprod-elastic-client.key and nprod-elastic-client.pem for the http server TLS configuration:

xpack.security.http.ssl.key:  /etc/elasticsearch/ssl/nprod-elastic-client.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/ssl/nprod-elastic-client.pem

and for the client TLS configuration as you use it in curl like below:

curl -XGET 'https://nprod-elastic:443/_cluster/health?pretty=true' --key /etc/elasticsearch/ssl/nprod-elastic-client.key --cert /etc/elasticsearch/ssl/nprod-elastic-client.pem --cacert /etc/elasticsearch/ssl/cacert

and you shouldn't. The client ( each client that should connect to Elasticsearch ) needs to have it's own client certificate and not reuse the one that the server uses. I hope this is clearer now.

Also in your final configuration you commented out a required setting

#xpack.security.http.ssl.client_authentication: optional

you need to re-enable this for the PKI realm to work as it is stated in our PKI realm docs

vijayakrishna.rg · August 8, 2019, 9:50pm

Thanks for taking time and providing explanation.

If i understand correctly, then do i need to have 3 certificates?

transport layer cert
http layer cert
client certs

that client cert point to pki so that we can use that cert while running curl command?

xpack:
 security:
   authc:
     realms:
       pki1:
         type: pki
         certificate_authorities: [ "/etc/elasticsearch/ssl/<place client cert here>" ]

vijayakrishna.rg · August 8, 2019, 11:04pm

i have created new set of certificates.

nprod-elastic-http-client.key
nprod-elastic-http-client.pem
cacert

i used openssl command to generate p12 and then jks using above certs and pointed that jks to pki.

xpack.security.http.ssl.client_authentication: optional
xpack:
  security:
    authc:
      realms:
        pki1:
          type: pki
          truststore:
            path: "/etc/elasticsearch/ssl/elastic-http-client.jks"
            password: "changeme"

openssl pkcs12 -export -in nprod-elastic-http-client.pem -inkey nprod-elastic-http-client.key -out elastic-http-client.p12 -name elastic -CAfile cacert -caname root

keytool -importkeystore -destkeystore elastic-http-client.jks -srckeystore elastic-http-client.p12 -srcstoretype pkcs12 -alias elastic

am now using above three certs while running curl command still having same issue.

curl -XGET 'https://nprod-elastic:443/_cluster/health?pretty=true' --key /etc/elasticsearch/ssl/nprod-elastic-http-client.key --cert /etc/elasticsearch/ssl/nprod-elastic-http-client.pem --cacert /etc/elasticsearch/ssl/cacert

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication token for REST request [/_cluster/health? pretty=true]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  }
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_cluster/health?pretty=true]",
"header" : {
  "WWW-Authenticate" : [
    "Bearer realm=\"security\"",
    "ApiKey",
    "Basic realm=\"security\" charset=\"UTF-8\""
  ]
}
  },
  "status" : 401
}`

ikakavas · August 9, 2019, 6:43am

That goes a little into consultative territory but the short version is that the best practice for this would be to have:

1 key / certificate pair for the http layer on Elasticsearch (xpack.security.http.ssl.key and xpack.security.http.ssl.certificate)
1 key / certificate pair ( all signed by the same CA - preferrably not the CA you use for everything else in your organization ) for each node to use for the transport layer.
1 key / certificate pair for each of the clients that should connect to Elasticsearch, so that they can use it with curl or any other client software.

vijayakrishna.rg:

that client cert point to pki so that we can use that cert while running curl command?
xpack:
 security:
   authc:
     realms:
       pki1:
         type: pki
         certificate_authorities: [ "/etc/elasticsearch/ssl/<place client cert here>" ]

No. The certificate_authorities there should point to the CA certificate that has signed the client certificate that curl will use to connect. Not the client certificate itself.

ikakavas · August 9, 2019, 6:48am

Did you change this as suggested above ?

ikakavas:

Also in your final configuration you commented out a required setting
#xpack.security.http.ssl.client_authentication: optional
you need to re-enable this for the PKI realm to work as it is stated in our PKI realm docs

You don't need to do that. If I understand this correctly, you have signed nprod-elastic-http-client.pem with the CA in cacert so the PKI Realm only needs to know about cacert and not about the actual certificate and key so

xpack:
 security:
   authc:
     realms:
       pki1:
         type: pki
         certificate_authorities: [ "/etc/elasticsearch/ssl/cacert" ]

should be enough.

system · September 6, 2019, 6:58am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.