How to authenticate user to elasticsearch based on PKI

kartmsb · June 4, 2019, 8:52am

Hi I am newbie to elasticsearch. And I was trying to enable security and PKI based authentication in elasticsearch. I am using 6.5.1 es. And following

"https://www.elastic.co/blog/elasticsearch-security-configure-tls-ssl-pki-authentication" this tutorial for setting up.

I have followed steps till creating client certificate . My requirement is authenticate user to elasticsearch with PKI . In above tutorial it is mentioned configuring kibana to authenticate to es cluster. Can't I use my own client instead of kibana? If so what is the procedure?

ikakavas · June 4, 2019, 8:56am

The procedure is the same exactly. For example there are details and examples on how to authenticate with a client key /certificate when using curl in section PKI Authentication in the blog post you mentioned.

kartmsb · June 4, 2019, 9:25am

Thanks for quick reply.

I followed the curl command curl "https://localhost:9200/_xpack/security/_authenticate?pretty
--key client.key --cert client.cer --cacert client-ca.cer -k -v " . mentioned in the blog.

But i ran into some error.
I had to make changes as --cacert instead of --cert in the above command to make it work.Is it okay with this change.What might be the reason for this change.?

After the just above change I again ran into one more error as below.
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : [
"Bearer realm="security"",
"Basic realm="security" charset="UTF-8""
]
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : [
"Bearer realm="security"",
"Basic realm="security" charset="UTF-8""
]
}
},
"status" : 401
}

To get rid of this error I had to make change to above command as

"curl -u "elastic":"elastic" https://localhost:9200/_xpack/security/_authenticate?pretty --key client.key --cacert client.cer --cacert client-ca.cer -k -v"

Why i need to pass elastic username/password ?.
In the above blog there is section to set built-in pwd to built-in user (I have done this ) but why we need to set this?
Is it because of above step I need to pass elastic username/pwd.

ikakavas · June 4, 2019, 9:44am

It looks like you haven't configured Elasticsearch correctly for PKI authentication. Please share the relevant parts of elasticsearch.yml.

kartmsb · June 4, 2019, 9:54am

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.authc.realms.pki1.type: pki

Fyi I am following steps exactly in the blog to get complete idea.

ikakavas · June 4, 2019, 11:02am

What is the error you ran into? What were the error message, what was in the logs. You'd need to provide some additional information for us to be able to assist you.

Do you mean that you changed --cert to --cacert ? That means that curl didn't use the Certificate to authenticate to elasticsearch and would explain the error you got:

You are trying to authenticate with a client certificate. If you don't send that client certificate (which is what happens when you don't use --cert in Curl, then it is expected that authentication fails.

You don't need to. What happens is that since you're not sending a client certificate and you are sending a username and password instead, Elasticsearch assumes that you want to authenticate with that username and password and allows you to do so. This is allowed since you are using xpack.security.http.ssl.client_authentication: optional which means that Elasticsearch will prompt for client certificate authentication but will also allow other means ( like basic auth ) if the client certificate authentication fails.

In summary:

Remove -u "elastic":"elastic" from the Curl command you try with
Change --cacert client.cer to --cert client.cer as it is described in the docs and the blog post.
If the above fails, share with us the exact error response and logs from Elasticsearch

kartmsb · June 4, 2019, 12:12pm

This is what ouput looks like after executing above command.

$ curl https://localhost:9200/_xpack/security/_authenticate?pretty --key client.key --cert client.cer --cacert client-ca.cer -k -v

* STATE: INIT => CONNECT handle 0x66f160; line 1404 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => WAITRESOLVE handle 0x66f160; line 1440 (connection #0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::1...
* TCP_NODELAY set
* STATE: WAITRESOLVE => WAITCONNECT handle 0x66f160; line 1521 (connection #0)
* Connected to localhost (::1) port 9200 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x66f160; line 1573 (connection #0)
* Marked for [keep alive]: HTTP default
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* ignoring certificate verify locations due to disabled peer verification
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x66f160; line 1587 (connection #0)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{ [5 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [81 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [1665 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
{ [59 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
} [891 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
} [264 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* TLSv1.2 (IN), TLS alert, Server hello (2):
{ [2 bytes data]
* error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
* Marked for [closure]: Failed HTTPS connection
* multi_done
* stopped the pause stream!
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
* Closing connection 0
* The cache now contains 0 members
* Expire cleared
curl: (35) error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown`Preformatted text`

kartmsb · June 4, 2019, 12:14pm

`Preformatted text`[2019-06-04T17:41:31,383][INFO ][o.e.c.r.a.AllocationService] [iLFF8Lv] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[bank][3]] ...]).
[2019-06-04T17:41:35,011][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [iLFF8Lv] caught exception while handling client http traffic, closing connection [id: 0xb3b0ea8d, L:0.0.0.0/0.0.0.0:9200 ! R:/0:0:0:0:0:0:0:1:64722]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1421) ~[?:1.8.0_45]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:1.8.0_45]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:1.8.0_45]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:1.8.0_45]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_45]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        ... 15 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_45]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:1.8.0_45]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:1.8.0_45]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_45]
        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1848) ~[?:1.8.0_45]
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:222) ~[?:1.8.0_45]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:969) ~[?:1.8.0_45]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:909) ~[?:1.8.0_45]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:906) ~[?:1.8.0_45]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_45]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1359) ~[?:1.8.0_45]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1457) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1365) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        ... 15 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[?:1.8.0_45]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_45]
        at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_45]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_45]

kartmsb · June 4, 2019, 12:15pm

Above given is the client output and elasticsearch ouput after executing the command

ikakavas · June 4, 2019, 12:26pm

Please use </> or backticks (```) to format your posts when you add logs/outputs/code/configuration. It makes it so much easier to read and will highly benefit the chances of someone reading through it and offering any help! It'd be great if you can edit your previous posts to fix that.

kartmsb · June 4, 2019, 12:30pm

Edited.

ikakavas · June 4, 2019, 12:32pm

Nothing has changed. You can use the preview panel on the right of the post when you edit it to see how it will look like in the end.

kartmsb · June 4, 2019, 12:36pm

oh sorry for this.

Edited can you check now.

TimV · June 5, 2019, 4:17am

These indicate that Elastisearch doesn't trust the CA that issued your client certificate.

Does this PKCS#12 file contain your client-ca.cer trusted certificate? That seems to be the problem here.

kartmsb · June 5, 2019, 5:05am

Hi ,
I have followed below command to generate certificate as per mentioned in the blog.
bin/elasticsearch-certutil ca
ENTER ENTER
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
ENTER ENTER ENTER

And placed it in the certs folder in config of elasticsearch



Followed by  below command to generate client certificate
bin/elasticsearch-certutil cert --ca \
config/certs/elastic-stack-ca.p12 \
-name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
ENTER
client.p12 ENTER
ENTER

So how to verify that PKCS#12 contains client-ca.cer trusted certificate??if it does not contain how can I add?

Please help.

TimV · June 5, 2019, 6:58am

How did you generate certs/elastic-certificates.p12 ?

That's the file that you are using as a truststore in xpack.security.http.ssl.truststore.path, it needs to have been generated using exactly the same CA that generated your client certificate.

kartmsb · June 5, 2019, 7:37am

This is the way I have generated elastic-certificates.p12

1.bin/elasticsearch-certutil ca (this will generate "elastic-stack-ca.p12" file for me ).
2.bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 (this will generate "elastic-certificates.p12" using "elastic-stack-ca.p12" from above step)

and my elastcsticsearch file truststore path configured as below

xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12

And I have used same CA certificate ("elastic-stack-ca.p12") for generating client certificate as per below command in the blog.

bin/elasticsearch-certutil cert --ca config/certs/elastic-stack-ca.p12 -name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
ENTER
client.p12 ENTER
ENTER

you mean to say I should use "config/certs/elastic-certificates.p12`" instead of config/certs/elastic-stack-ca.p12 for generating client certificate?

TimV · June 7, 2019, 3:19am

I'm not sure what's happened here, but something isn't quite right. If you did exactly what you described then you shouldn't get this error.

Can you run these commands:

keytool -list -storepass "" -keystore config/certs/elastic-stack-ca.p12
keytool -list -storepass "" -keystore config/certs/elastic-certificates.p12
keytool -list -storepass "" -keystore client.p12

system · July 5, 2019, 3:19am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.