Full verification mode possible with self generated CA?

tombar · July 29, 2019, 1:30pm

I'm just getting started with securing Elastic Stack. My setup is a cloud VM where both Elasticsearch and Kibana are running. It is a one node setup right now.

I generated a CA with certutil, and then generated a pkcs12 key for my Elasticsearch node using this CA.
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name my-server01 --ip 192.168.0.1 (placeholder) --dns my-server01.vpc.com

I'v got Elasticsearch running with full host verification. Now I'm adding Kibana to the mix by converting the pkcs12 key from the CA to a pem file and it's working fine with elasticsearch.ssl.verificationMode: certificate. However once I set verificationMode to full Kibana can't discover Elasticsearch anymore.

Isn't it possible to use full verification mode with a self generated CA or am I doing something wrong here?

ikakavas · July 29, 2019, 2:10pm

Hi, you need to give us a little more information. Relevant parts of your kibana.yml file would be a good start !

How did you go about doing the following ?

and what setting in Kibana did you use the resulting PEM for ?

What errors do you see in the Elasticsearch logs or the Kibana logs ?

tombar · July 29, 2019, 6:16pm

I was using OpenSSL as I couldn't create a PEM with certutil.
openssl pkcs12 -in CertificateName.p12 -out CertificateName.pem -nodes

and what setting in Kibana did you use the resulting PEM for ?

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/my-server01.pem" ]

I tried to both put the PEM from the Elasticsearch node and from the elastic-stack-ca.pkcs12 here, neither works.

What errors do you see in the Elasticsearch logs or the Kibana logs ?

Jul 29 12:56:09 my-server01 kibana[18358]: {"type":"log","@timestamp":"2019-07-29T12:56:09Z","tags":["warning","elasticsearch","admin"],"pid":18358,"message":"Unable to revive connection https://my-server01:9200/"}
Jul 29 12:56:09 my-server0101 kibana[18358]: {"type":"log","@timestamp":"2019-07-29T12:56:09Z","tags":["warning","elasticsearch","admin"],"pid":18358,"message":"No living connections"}

ikakavas · July 29, 2019, 7:54pm

openssl pkcs12 -in CertificateName.p12 -out CertificateName.pem -nodes

probably only got the server certificate out, and not the ca one. You can verify which certficate you got with

openssl x509 -in CertificateName.pem -text -noout

You might need to run

openssl pkcs12 -in CertificateName.p12 -out CertificateName.pem -nodes -cacerts

depending on what you see in the previous command.

You can, unfortunately, only use PEM files for kibana, so setting the PKCS#12 store wouldn't work.

That error ( if it;s the only one ) seems to indicate that Elasticsearch is not up and running. What are the content of the elasticsearch logs. Any errors there? Can you verify that Elasticsearch does indeed start ?

tombar · July 30, 2019, 1:30pm

I think I've got both. However there's an unsupported tag 6, maybe that's a problem?

Bag Attributes
    friendlyName: my-server01
    localKeyID: 123456789
subject=CN = my-server01

issuer=CN = Elastic Certificate Tool Autogenerated CA

-----BEGIN CERTIFICATE-----
XYZ
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: ca
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=CN = Elastic Certificate Tool Autogenerated CA

issuer=CN = Elastic Certificate Tool Autogenerated CA

-----BEGIN CERTIFICATE-----
XYZ
-----END CERTIFICATE-----

You can, unfortunately, only use PEM files for kibana, so setting the PKCS#12 store wouldn't work.

Sorry, I meant I tried to use the converted PEM of the elastic-stack-ca.pkcs12, but as that is missing the server information this won't work in full verification mode I guess.

That error ( if it;s the only one ) seems to indicate that Elasticsearch is not up and running. What are the content of the elasticsearch logs. Any errors there? Can you verify that Elasticsearch does indeed start ?

Elasticsearch is definitely running, I get correct responses to curl commands.

ikakavas · July 30, 2019, 1:41pm

You can just leave the last part in the PEM file:

-----BEGIN CERTIFICATE-----
XYZ
-----END CERTIFICATE-----

that's all you need. If this doesn't solve your problem, then I'd need to see more of your kibana logs. there should be something more apart from the line you shared.

What is the curl command you are running ?

tombar · July 30, 2019, 1:56pm

You are right:

Jul 30 13:49:38 my-server01 kibana[21691]: {"type":"log","@timestamp":"2019-07-30T13:49:38Z","tags":
["status","plugin:ui_metric@7.2.0","info"],"pid":21691,"state":"green",
"message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 30 13:49:39 my-server01 kibana[21691]: {"type":"log","@timestamp":"2019-07-30T13:49:39Z",
"tags":["error","elasticsearch","admin"],"pid":21691,"message":"Request error, 
retrying\nGET https://my-server01:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip 
=> Hostname/IP does not match certificate's altnames: Host: my-server01. is not in the cert's altnames: 
IP Address:[replaced host ip address], DNS:my-server01.privatecloud.com"}

It seems I made an error when generating the server certificate.

What is the curl command you are running ?

I'm just running a GET on Elasticsearch's Port with basic authentication with elastic user. I'm actually using Postman and am converting the curl commands as documented.

ikakavas · July 30, 2019, 2:22pm

Nice! Not that you made an error but that it looks like your issue might have a simple solution.

The only reason I'm asking you to share them is for us to get a mutual understanding of how the setup is and what errors you get. i.e. , if you run

curl -k -u elastic:password https://IP_ADDRESS:port

and this is successful, it doesn't mean much if your kibana settings say i.e.

elasticsearch.url: https://FDQN:port
elasticsearch.username: kibana
elasticsearch.password: otherpassword

because your curl command checks with a different user, without validating the certificate the server presents and using the IP address to connect to the Elasticsearch server instead of the IP address ( that latter might also affect the certificate validation ). It's always better if you can share the exact command and the exact output/error/log , rather than a description of them.

tombar · July 30, 2019, 2:27pm

Yes, I see your point. Unfortunately I have to anonymize a lot of things which makes it a little annoying to share logs.

I'll have a look at how I generated the certificate again, I'm not sure right now where the error is.

tombar · July 30, 2019, 2:30pm

Well, your last suggestion did it!

I didn't know I had to use the FQDN in kibana.yml. Now everything is working perfectly! Thank you very much for your support.

So should I use the FQDN for Elasticsearch for every part of Elastic Stack? Also the beats? They don't seem to use full host verification however...

ikakavas · July 30, 2019, 2:43pm

You don't have to. The other two possibilities would be to

a. Add the IP address as a SAN in the certificate
b. Disable the hostname verification ( not actually suggesting it as a good idea )

Same goes for beats or other stack products talking to Elasticsearch (or any other external system talking to any server over TLS with hostname verification on). Either add the IP Address of the server as a SAN in the the certificate and connect to the IP address, or connect to the FQDN (which I assume is already a SAN in the server certificate )

system · August 27, 2019, 2:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.