Using the Java Keystore with Shield

Elastic Stack Elasticsearch
1

Hi so I'm trying to connect Kibana to Elasticsearch both of which have the Shield Plugin installed and both are running at the moment of this testing on the same machine. I have followed the guides from elastic: https://www.elastic.co/guide/en/shield/current/kibana.html, https://www.elastic.co/guide/en/shield/current/ssl-tls.html#generate-csr. I was able to use the keytool to generate a certificate and private key into the keystore and then used both to create a CSR which I submitted to Comodo and received back a signed certificate. I then followed instructions and downloaded the X509, Base64 encoded certificate and imported it into the keystore. The issue that I am running into is in the Kibana.yml file, its requesting the path for server.ssl.cert and server.ssl.key. I have tried pointing it to the keystore and pointing the server.ssl.cert to the downloaded cert and both have yield no results. I am wondering if the server.ssl.key is not pointed correctly. Where would the key reside if not the keystore in this case? Also do I have to worry about the elasticsearch.ssl.ca option too in the Kibana.yml? Should that be pointing to the keystore as well? Lastly, for setting the shield.encryptionKey is this something that I make up or should I put the password for the keystore here? Thanks for all the help in advance!

Also here is a copy of the log file for Kibana:
FATAL { [Error: shield.encryptionKey is required in kibana.yml.]
cause: [Error: shield.encryptionKey is required in kibana.yml.],
isOperational: true }
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]
FATAL { [Error: HTTPS is required. Please set server.ssl.key and server.ssl.cert in kibana.yml.]
cause: [Error: HTTPS is required. Please set server.ssl.key and server.ssl.cert in kibana.yml.],
isOperational: true }
FATAL { [Error: EACCES, permission denied '/home//<cert_name.crt>']
errno: -13,
code: 'EACCES',
path: '/home//<cert_name.crt>',
syscall: 'open' }
FATAL { [Error: EACCES, permission denied '/home//<cert_name.cer>']
errno: -13,
code: 'EACCES',
path: '/home//<cert_name.cer>',
syscall: 'open' }
FATAL { [Error: EACCES, permission denied '/home//<cert_name.crt>']
errno: -13,
code: 'EACCES',
path: '/home//<cert_name.crt>',
syscall: 'open' }
FATAL { [Error: EACCES, permission denied '/home//<cert_name.crt>']
errno: -13,
code: 'EACCES',
path: '/home//<cert_name.crt>',
syscall: 'open' }
FATAL { [Error: EACCES, permission denied '/home//<cert_name.cer>']
errno: -13,
code: 'EACCES',
path: '/home//<cert_name.cer>',
syscall: 'open' }
FATAL { [Error: HTTPS is required. Please set server.ssl.key and server.ssl.cert in kibana.yml.]
cause: [Error: HTTPS is required. Please set server.ssl.key and server.ssl.cert in kibana.yml.],
isOperational: true }

2

Can you show us the relevant KB config sections?

4

# If your Elasticsearch is protected with basic auth, these are the user credentials

# used by the Kibana server to perform maintenance on the kibana_index at startup. Your Kibana

# users will still need to authenticate with Elasticsearch (which is proxied through

# the Kibana server)

elasticsearch.username: "user"

elasticsearch.password: "pass"

# SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

server.ssl.cert: /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts

server.ssl.key: /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts

# Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

# elasticsearch.ssl.cert: /path/to/your/client.crt

# elasticsearch.ssl.key: /path/to/your/client.key

# If you need to provide a CA certificate for your Elasticsearch instance, put

# the path of the pem file here.

elasticsearch.ssl.ca: /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts

shield.encryptionKey: "something_secret"

5

I don't think that is valid is it?

6

Hi Kevin,

To add to what Mark said, with Kibana you cannot use the Java keystore. Kibana is also an HTTP server so you will need a private key elasticsearch.ssl.key and certificate elasticsearch.ssl.cert for Kibana to serve content using TLS. The elasticsearch.ssl.ca setting is a path that points to CA certificate that was used to sign the certificates of the elasticsearch node that Kibana will be communicating with.

Some more detail are available in the Kibana docs: https://www.elastic.co/guide/en/kibana/current/production.html#enabling-ssl

7

Hey @warkolm and @jaymode, thanks for the info. I started getting that feeling going through this again this week so I thought I would try to get it working without using the keystore. Thanks for confirming that! I'm trying to get it working, but now I'm running into an issue. Have you guys seen this error before?

{"type":"log","@timestamp":"2016-03-08T23:02:39+00:00","tags":["warning","elasticsearch"],"pid":1621,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2016-03-08T23:02:39+00:00","tags":["warning","elasticsearch"],"pid":1621,"message":"No living connections"}
{"type":"log","@timestamp":"2016-03-08T23:02:42+00:00","tags":["warning","elasticsearch"],"pid":1621,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2016-03-08T23:02:42+00:00","tags":["warning","elasticsearch"],"pid":1621,"message":"No living connections"}
{"type":"ops","@timestamp":"2016-03-08T23:02:44+00:00","tags":[],"pid":1621,"os":{"load":[0.0029296875,0.0263671875,0.04541015625],"mem":{"total":2095730688,"free":679718912},"uptime":6153993},"proc":{"uptime":5088.769,"mem":{"rss":144424960,"heapTotal":126609920,"heapUsed":91065680},"delay":0.4524869918823242},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 86.8MB uptime: 1:24:49 load: [0.00 0.03 0.05] delay: 0.452"}

8

Also, I am assuming Logstash doesn't utilize the keystore either? Just Elasticsearch utilizes the Java Keystore??

9

Logstash will use a keystore if you are using the transport protocol. When using the HTTP protocol the configuration only requires a path to the CA cert.

Regarding the error you are seeing, what does your logstash configuration look like?

10

Okay. I fixed the error, it was with regards to Kibana. I believe in the elasticsearch url, I set it to be https instead of http. I got Kibana working with shield plugin and ssl is enabled. The frontend login screen is awesome! I am wondering though if I have two kibana instances and they are behind a netscaler load balancers, if I put a certificate on the load balancer and not on Kibana, would Kibana detect the load balancer cert and work or do I need to keep the certificate on Kibana and just route the loadbalancer through port 443?

11

With the current release you need to route the loadbalancer to kibana over SSL because of the check the shield plugin has. We're adding a feature to allow this check to be disabled, but the loadbalancer must serve HTTPS to the clients or the session will not work since the cookie has the secure flag set.

12

@jaymode do you have any idea why when I change the elastisearch.url in the kibana.yml from http to https I get that living connection error above??

13

when you change the url, do you enable ssl on the cluster? any log messages on the elasticsearch side of things?

14

Ah I figured it out! As soon as I commented out the elasticsearch.ssl.ca option it worked! So now I am fully encrypted. Thanks for all the help in answering my questions!

15

Ah @jaymode sorry one more question. Does Marvel work with keystores and if so can I point it at the same keystore that my elasticsearch instance is pointed too?

edit: also do I need to configure Marvel on each and every node that has the marvel plugin installed or can I just configure the one on Kibana? Thanks!

16

Are you exporting marvel data locally or to a remote cluster?

If you are exporting locally (using a exporter with type: local or the default one), then you do not need to worry about SSL configuration since it is communicating internally with the cluster.

If you are exporting data to a remote cluster, then yes you will need to use a truststore/keystore; this store needs to contain the certificates necessary to trust the remote nodes that you are exporting to. If both clusters use the same CA, then you should be able to reuse the existing keystore as a truststore.

For details on the settings check out https://www.elastic.co/guide/en/marvel/current/configuration.html