Full verification mode possible with self generated CA?

Elastic Stack Kibana
1

I'm just getting started with securing Elastic Stack. My setup is a cloud VM where both Elasticsearch and Kibana are running. It is a one node setup right now.

I generated a CA with certutil, and then generated a pkcs12 key for my Elasticsearch node using this CA.
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name my-server01 --ip 192.168.0.1 (placeholder) --dns my-server01.vpc.com

I'v got Elasticsearch running with full host verification. Now I'm adding Kibana to the mix by converting the pkcs12 key from the CA to a pem file and it's working fine with elasticsearch.ssl.verificationMode: certificate. However once I set verificationMode to full Kibana can't discover Elasticsearch anymore.

Isn't it possible to use full verification mode with a self generated CA or am I doing something wrong here?

2

Hi, you need to give us a little more information. Relevant parts of your kibana.yml file would be a good start !

How did you go about doing the following ?

and what setting in Kibana did you use the resulting PEM for ?

What errors do you see in the Elasticsearch logs or the Kibana logs ?

3

I was using OpenSSL as I couldn't create a PEM with certutil.
openssl pkcs12 -in CertificateName.p12 -out CertificateName.pem -nodes

and what setting in Kibana did you use the resulting PEM for ?

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/my-server01.pem" ]

I tried to both put the PEM from the Elasticsearch node and from the elastic-stack-ca.pkcs12 here, neither works.

What errors do you see in the Elasticsearch logs or the Kibana logs ?

Jul 29 12:56:09 my-server01 kibana[18358]: {"type":"log","@timestamp":"2019-07-29T12:56:09Z","tags":["warning","elasticsearch","admin"],"pid":18358,"message":"Unable to revive connection https://my-server01:9200/"}
Jul 29 12:56:09 my-server0101 kibana[18358]: {"type":"log","@timestamp":"2019-07-29T12:56:09Z","tags":["warning","elasticsearch","admin"],"pid":18358,"message":"No living connections"}

4 
openssl pkcs12 -in CertificateName.p12 -out CertificateName.pem -nodes

probably only got the server certificate out, and not the ca one. You can verify which certficate you got with

openssl x509 -in CertificateName.pem -text -noout

You might need to run

openssl pkcs12 -in CertificateName.p12 -out CertificateName.pem -nodes -cacerts

depending on what you see in the previous command.

You can, unfortunately, only use PEM files for kibana, so setting the PKCS#12 store wouldn't work.

That error ( if it;s the only one ) seems to indicate that Elasticsearch is not up and running. What are the content of the elasticsearch logs. Any errors there? Can you verify that Elasticsearch does indeed start ?

5

I think I've got both. However there's an unsupported tag 6, maybe that's a problem?

Bag Attributes
    friendlyName: my-server01
    localKeyID: 123456789
subject=CN = my-server01

issuer=CN = Elastic Certificate Tool Autogenerated CA

-----BEGIN CERTIFICATE-----
XYZ
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: ca
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=CN = Elastic Certificate Tool Autogenerated CA

issuer=CN = Elastic Certificate Tool Autogenerated CA

-----BEGIN CERTIFICATE-----
XYZ
-----END CERTIFICATE-----

You can, unfortunately, only use PEM files for kibana, so setting the PKCS#12 store wouldn't work.

Sorry, I meant I tried to use the converted PEM of the elastic-stack-ca.pkcs12, but as that is missing the server information this won't work in full verification mode I guess.

That error ( if it;s the only one ) seems to indicate that Elasticsearch is not up and running. What are the content of the elasticsearch logs. Any errors there? Can you verify that Elasticsearch does indeed start ?

Elasticsearch is definitely running, I get correct responses to curl commands.

6

You can just leave the last part in the PEM file:

-----BEGIN CERTIFICATE-----
XYZ
-----END CERTIFICATE-----

that's all you need. If this doesn't solve your problem, then I'd need to see more of your kibana logs. there should be something more apart from the line you shared.

What is the curl command you are running ?

7

You are right:

Jul 30 13:49:38 my-server01 kibana[21691]: {"type":"log","@timestamp":"2019-07-30T13:49:38Z","tags":
["status","plugin:ui_metric@7.2.0","info"],"pid":21691,"state":"green",
"message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Jul 30 13:49:39 my-server01 kibana[21691]: {"type":"log","@timestamp":"2019-07-30T13:49:39Z",
"tags":["error","elasticsearch","admin"],"pid":21691,"message":"Request error, 
retrying\nGET https://my-server01:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip 
=> Hostname/IP does not match certificate's altnames: Host: my-server01. is not in the cert's altnames: 
IP Address:[replaced host ip address], DNS:my-server01.privatecloud.com"}

It seems I made an error when generating the server certificate.

What is the curl command you are running ?

I'm just running a GET on Elasticsearch's Port with basic authentication with elastic user. I'm actually using Postman and am converting the curl commands as documented.

8

Nice! Not that you made an error :slight_smile: but that it looks like your issue might have a simple solution.

The only reason I'm asking you to share them is for us to get a mutual understanding of how the setup is and what errors you get. i.e. , if you run

curl -k -u elastic:password https://IP_ADDRESS:port

and this is successful, it doesn't mean much if your kibana settings say i.e.

elasticsearch.url: https://FDQN:port
elasticsearch.username: kibana
elasticsearch.password: otherpassword

because your curl command checks with a different user, without validating the certificate the server presents and using the IP address to connect to the Elasticsearch server instead of the IP address ( that latter might also affect the certificate validation ). It's always better if you can share the exact command and the exact output/error/log , rather than a description of them.

9

Yes, I see your point. Unfortunately I have to anonymize a lot of things which makes it a little annoying to share logs.

I'll have a look at how I generated the certificate again, I'm not sure right now where the error is.

10

Well, your last suggestion did it! :smile:

I didn't know I had to use the FQDN in kibana.yml. Now everything is working perfectly! Thank you very much for your support.

So should I use the FQDN for Elasticsearch for every part of Elastic Stack? Also the beats? They don't seem to use full host verification however...

11

You don't have to. The other two possibilities would be to

a. Add the IP address as a SAN in the certificate
b. Disable the hostname verification ( not actually suggesting it as a good idea )

Same goes for beats or other stack products talking to Elasticsearch (or any other external system talking to any server over TLS with hostname verification on). Either add the IP Address of the server as a SAN in the the certificate and connect to the IP address, or connect to the FQDN (which I assume is already a SAN in the server certificate )

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.