# Can't configure PKI authentication **URL:** \ **Category:** Elasticsearch\ **Created:** [June 8, 2017, 7:28am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673 "2017-06-08T07:28:59Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 7:28am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/1 "2017-06-08T07:28:59Z")
Hey all, I'm trying to use PKI authentication in my cluster, meaning the client will need to present certificates in order to get data. I wrote the following in elasticsearch.yml: `xpack: security: authc: realms: pki1: type: pki` And removed my "native" realm. I changed nothing else in my code (c#). Right now, I can still get data using my "elastic" user, and if I don't use it I get`"missing authentication token for REST request".` Unless I don't understand something, this seems wrong, as I'm expecting a security exception of some sort. Thank you! EDIT: I tried setting xpack.security.http.ssl.client\_authentication: required. This changes everything. Now I'm getting: `'The underlying connection was closed: An unexpected error occurred on a send.'` Which is good, I think. I generated a certificate for my client using certgen, and used it to generate an x509Certificate in C#, I also added the autogenerated CA that signed it to xpack.ssl.certificate\_authorities, and mapped the CN to a role. Still getting this error. Installing the client's CA on the elastic machine didn't help either. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 8, 2017, 7:59am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/2 "2017-06-08T07:59:37Z")
Hi @Shlomi_Toren Where possible please use the code button (``) when pasting in your config files so that it retains the correct formatting. Thanks. There's a few things going on here: - The `elastic` user isn't part of the `native` realm, so removing that realm won't change whether the elastic user works. ([More Info](https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html#built-in-users)) - Simply enabling a PKI realm doesn't _force_ that users need to use TLS/PKI. If you want to enforce client certificates you need to do that at the http layer with the `xpack.security.http.ssl.client_authentication` setting . ([More Info](https://www.elastic.co/guide/en/x-pack/current/security-settings.html#http-tls-ssl-settings)) - You say _I'm expecting a security exception_ but the error message you received _(missing authentication token for REST request)_ **is** a security exception (on the Elasticsearch side, I haven't looked at the C# side) . Can you clarify how that message doesn't meet your expectation? A working configuration for PKI would typically look something like this: ```auto xpack.security.authc.realms: pki1: type: pki order: 0 certificate_authorities: tls/client/ca.crt files.role_mapping: pki-roles.yml xpack.ssl: certificate: tls/server/node01/node01.crt key: tls/server/node01/node01.key xpack.security.http.ssl: enabled: true client_authentication: optional certificate_authorities: ["tls/client/ca.crt"] xpack.security.transport.ssl: enabled: true certificate_authorities: ["tls/server/ca/ca.crt"] ``` That said, there's lots of different configuration options, so your config will almost certainly look different, but that's the rough approach. ---
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 8:10am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/3 "2017-06-08T08:10:39Z")
I'll try this and see if it works, thanks. Just to clarify: The fact that the `elastic` user is NOT a part of the `native` realm, means I HAVE to use it with PKI? ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 8, 2017, 8:14am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/4 "2017-06-08T08:14:20Z")
`xpack.ssl.certificate` is the certificate that the server provides to incoming connections (either HTTP or transport). You can't simply use a CA for that purpose as the server needs to have its own certificate configured in some way. You can use a keystore instead of a certificate, but you need one of them. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 8, 2017, 8:17am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/5 "2017-06-08T08:17:12Z")
> [@Shlomi\_Toren](#): > > Just to clarify: The fact that the elastic user is NOT a part of the native realm, means I HAVE to use it with PKI? You don't **have** to use it. But removing the native realm doesn't change whether that user is available or not. As a general rule your code shouldn't ever use the elastic user. It's a superuser, roughly equivalent to `root` on a unix system. Your code should be using it's own user (either representing the person logged in, or the application itself, depending on your security mode). Whether you authenticate that user via password or TLS certificate is up to you. ---
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 8:18am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/6 "2017-06-08T08:18:28Z")
> [@TimV](#): > > `xpack.ssl.certificate` is the certificate that the server provides to incoming connections (either HTTP or transport). > > You can't simply use a CA for that purpose as the server needs to have its own certificate configured in some way. > > You can use a keystore instead of a certificate, but you need one of them. Of course, I confused it for something else. Sorry. ---
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 8:19am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/7 "2017-06-08T08:19:38Z")
> [@TimV](#): > > > [@Shlomi\_Toren](#): > > > > Just to clarify: The fact that the elastic user is NOT a part of the native realm, means I HAVE to use it with PKI? > > You don't **have** to use it. But removing the native realm doesn't change whether that user is available or not. > > As a general rule your code shouldn't ever use the elastic user. It's a superuser, roughly equivalent to `root` on a unix system. Your code should be using it's own user (either representing the person logged in, or the application itself, depending on your security mode). Whether you authenticate that user via password or TLS certificate is up to you. Ok, to be more specific: Do I need to specify a user in the request even if I'm using a certificate? And should that user be the CN of the certificate? ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 8, 2017, 8:24am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/8 "2017-06-08T08:24:39Z")
> [@Shlomi\_Toren](#): > > Do I need to specify a user in the request even if I'm using a certificate? And should that user be the CN of the certificate? No, if you authenticate via certificate you do not need to provide a user. For example this curl command (my apologies for not having a more windows-oriented example) will successfully authenticate a user using only a certificate (the password supplied is the passphrase on the certificate file, it's not sent to the server). ```auto curl --cert my-cert.p12:cert-password \ --cacert server/ca.crt \ 'https://localhost:9200/_xpack/security/_authenticate?pretty' ``` ---
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 8:25am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/9 "2017-06-08T08:25:43Z")
> [@TimV](#): > > Hi @Shlomi_Toren > > Where possible please use the code button (``) when pasting in your config files so that it retains the correct formatting. Thanks. > > There's a few things going on here: > > - The `elastic` user isn't part of the `native` realm, so removing that realm won't change whether the elastic user works. ([More Info](https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html#built-in-users)) > - Simply enabling a PKI realm doesn't _force_ that users need to use TLS/PKI. If you want to enforce client certificates you need to do that at the http layer with the `xpack.security.http.ssl.client_authentication` setting . ([More Info](https://www.elastic.co/guide/en/x-pack/current/security-settings.html#http-tls-ssl-settings)) > - You say _I'm expecting a security exception_ but the error message you received _(missing authentication token for REST request)_ **is** a security exception (on the Elasticsearch side, I haven't looked at the C# side) . Can you clarify how that message doesn't meet your expectation? > > A working configuration for PKI would typically look something like this: > > ```auto > xpack.security.authc.realms: > pki1: > type: pki > order: 0 > certificate_authorities: tls/client/ca.crt > files.role_mapping: pki-roles.yml > > xpack.ssl: > certificate: tls/server/node01/node01.crt > key: tls/server/node01/node01.key > > xpack.security.http.ssl: > enabled: true > client_authentication: optional > certificate_authorities: ["tls/client/ca.crt"] > > xpack.security.transport.ssl: > enabled: true > certificate_authorities: ["tls/server/ca/ca.crt"] > > ``` > > That said, there's lots of different configuration options, so your config will almost certainly look different, but that's the rough approach. Still didn't cut it. I guess I need to show you some code now, but the problem is I have an infrastructure for other types of REST applications that I'm using, and I'm using RESTSharp in it. Here's my code: ``` System.Net.NetworkCredential c = new System.Net.NetworkCredential("elastic", "password!"); httpRequest.Credentials = c; X509Certificate cert = new X509Certificate(@"path\client1.crt"); httpRequest.ClientCertificates = new X509CertificateCollection(); httpRequest.ClientCertificates.Add(cert); ``` The paths I'm using are correct, as I can generate the certificate, and the rest of this code works fine without PKI. Do you have an idea as to what could go wrong? ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 8, 2017, 8:30am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/10 "2017-06-08T08:30:11Z")
> [@Shlomi\_Toren](#): > > new X509Certificate(@"path\client1.crt"); What format is `client1.crt`? Typically a `.crt` file is only the public side of the certificate, and you need a separate key file that contains the private key. It's possible to have a combined PKCS-12 formatted key, but I'd double check your file formats. ---
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 8:32am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/11 "2017-06-08T08:32:04Z")
> [@TimV](#): > > > [@Shlomi\_Toren](#): > > > > new X509Certificate(@"path\client1.crt"); > > What format is `client1.crt`? Typically a `.crt` file is only the public side of the certificate, and you need a separate key file that contains the private key. > > It's possible to have a combined PKCS-12 formatted key, but I'd double check your file formats. I see. certgen generated this file along with a .key file. How do I use both of them in this context? Same goes for your curl command. I'm trying it and getting `curl: (58) unable to set private key file: 'client1.crt' type PEM` So I guess I do have a problem with the .crt file. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 8, 2017, 10:14am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/12 "2017-06-08T10:14:50Z")
I'm not a .NET dev, but I believe X509Certificate requires a combined certificate in PKCS-12 or pfx format. If you have openssl available you can combine the `.crt` and `.key` files into a `.p12` with: ```auto openssl pkcs12 -export -clcerts -in client1.crt -inkey client1.key -out client1.p12 ``` I think `pvk2pfx` does something similar on windows, but I can't offer any more advice than that. ---
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 11:07am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/13 "2017-06-08T11:07:37Z")
> [@TimV](#): > > I'm not a .NET dev, but I believe X509Certificate requires a combined certificate in PKCS-12 or pfx format. > > If you have openssl available you can combine the `.crt` and `.key` files into a `.p12` with: > > ```auto > openssl pkcs12 -export -clcerts -in client1.crt -inkey client1.key -out client1.p12 > > ``` > > I think `pvk2pfx` does something similar on windows, but I can't offer any more advice than that. Cool, I was looking for the openssl command but couldn't find it, thanks. I packaged the certificate and key, gave a password, and gave the `X509Certificate` object the p12 file and the password. Still didn't work. Can you think about anything else that might cause my issue? Maybe the certificate itself? I gave it the IP and DNS name of the client that's running it, but I don't know if ES even checks that. ---
**Author:** ![Shlomi\_Toren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shlomi_toren/32/18791_2.png) [@Shlomi\_Toren](https://discuss.elastic.co/u/Shlomi_Toren)\ **Post date:** [June 8, 2017, 11:14am UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/14 "2017-06-08T11:14:51Z")
Holy hell. Turns out, there's also a `X509Certificate2` object. It just worked. I guess that solves it. Thank you so much for your help TIm! ---
**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\ **Post date:** [June 8, 2017, 6:23pm UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/15 "2017-06-08T18:23:24Z")
Glad you got it sorted out @Shlomi_Toren 👍 We also have documentation for [working with certificates](https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/working-with-certificates.html#_client_certificates) with NEST, with a helper class, `ClientCertificate` to make it a little easier to set up on the client (.NET Framework only, as .NET Core does not allow the PrivateKey property to be set on the crypto service provider that is used to generate the pfx file at runtime.) > ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 6:23pm UTC](https://discuss.elastic.co/t/cant-configure-pki-authentication/88673/16 "2017-07-06T18:23:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.