SSL/TLS on ELK cluster

Elastic Stack Elasticsearch
41

Yes that is already configured in kibana.yml
elasticsearch.ssl.ca: /etc/elasticsearch/shield/cert.pem

42

What cert is that? Is it the intermediate CA cert?

43

No it is not intermediate cert.
Using following we generated cert.pem and I am using this cert-

Here are the steps we followed to generate the certificate -
1.Put the request using -

openssl req -nodes -newkey rsa:2048 -sha256 -keyout myserver.key -out server.csr
2.On the CA website we got the certificates available in following formats -

DER, CRT, PKCS7b

( When I download PKCS7b file , it takes .pem extension )

44

So it is the cert of the elasticsearch server? It should be the intermediate certificate that you use for elasticsearch.ssl.ca and you may have also include the root CA certificate as well like:

elasticsearch.ssl.ca: [ "/path/to/root.pem", "/path/to/intermediate.pem" ]

Please make sure these are in PEM format. You may need to use the openssl commands we used before if they are not

45

Yes it was cert of ES server. Ok , I have int. cert and root cert in der format which I converted to pem format using -
e.g.
openssl x509 -inform der -in caintermediatecert.der -out caintermediatecert.pem

Did for both int and root certs and configured as given by you. Kibana service is still not starting.

The kiabana log entries are 1 day old after that its not generating any logs -

{"type":"log","@timestamp":"2016-07-15T05:35:24+00:00","tags":["warning","elasticsearch"],"pid":1960,"message":"No living connections"}
{"type":"log","@timestamp":"2016-07-15T05:35:27+00:00","tags":["warning","elasticsearch"],"pid":1960,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2016-07-15T05:35:27+00:00","tags":["warning","elasticsearch"],"pid":1960,"message":"No living connections"}
46

So you get not output at all from running "bin/kibana"?

47

I am running kibana using /etc/init.d/kibana start/stop

If I start using /bin/kibana , I get following -

[root@irldxvm022 kibana]# bin/kibana serve
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]

48

This implies that one of your certificate/key files isn't really a PEM file or has extra text in it outside of the ----BEGIN and -----END lines. I suggest you inspect each file

49

Ok I checked both files but I do not see any spaces or extra characters as such. I converted those from der format to pem and used as it is.

We have few days left of Shield trial license.
To speed up the resolution , should I set up webex session. Is that fine with you ? Are you working in EST time zone ?

Thanks,
Vinod

50

Hi Jay,

Now we trying with the commands given in the official document-
We ran following commands -

keytool -genkey -alias elk01 -keystore elk01.jks -keyalg RSA -keysize 2048 -validity 712 -ext san=dns:irldxvm022.irl.in.ibm.com,ip:9.126.112.72

keytool -certreq -alias elk01 -keystore elk01.jks -file elk01.csr -keyalg rsa -ext san=dns:irldxvm022.irl.in.ibm.com,ip:9.126.112.72

After that we are uploading the csr file on the portal for signing , however we are getting the errors.
PFA is the screenshot of errors.

888×154 19 KB

51

Hi Vinod,

I am not sure what causes that. The keysize you specified has the proper minimum size. Did you use the same name when running keytool ("CN=9.126.112.72,OU=Research,L=New Delhi,ST=N/A,C=IN") ?

I think the portal administrators may be able to help you better as I am not familiar with it.

-Jay

52

If I use the openssl command to generate csr , its working. But again that is not helping us in any way, we have already tried that. Can you explain the first command mentioned in the - https://www.elastic.co/guide/en/shield/current/ssl-tls.html#install-signed-cert
i.e.

Create a node keystore and import your CA’s certificate with Java Keytool. This configures the node to trust certificates signed by the CA. For Elasticsearch to access the keystore, it must be located under the Elasticsearch configuration directory. For example, the following command creates a keystore for node01 and and imports the CA certificate cacert.pem.

cd CONFIG_DIR/shield
keytool -importcert -keystore node01.jks -file cacert.pem -alias my_ca
53

The command simply imports the CA certificate (cacert.pem) into node01.jks, which will be created if it does not exist. The certificate is also given an alias, which is not required.

54

Hi Jay,

Peter has sent us the document which helps to configure the self signed certs. I have carried out the steps but we are still getting errors.

Regards,
Vinod

55

Hi

Ok I have configured now only one instance of ELK and configured shield on it. My logstash and ES logs are clean and communication between them is working. If I run curl on https its giving me the o/p

But kibana is throwing error -

>   log   [10:23:12.369] [warning][elasticsearch] Unable to revive connection: https://127.0.0.1:9200/
>   log   [10:23:12.370] [warning][elasticsearch] No living connections
>   log   [10:23:12.374] [error][status][plugin:elasticsearch] Status changed from yellow to red - Unable to connect to Elasticsearch at https://127.0.0.1:9200.
>   log   [10:23:14.909] [warning][elasticsearch] Unable to revive connection: https://127.0.0.1:9200/
>   log   [10:23:14.911] [warning][elasticsearch] No living connections
>   ops   [10:23:15.722]  memory: 65.8MB uptime: 0:00:07 load: [0.74 0.56 0.52] delay: 3.218
> ^C
> [root@irldxvm074 kibana]# telnet 127.0.0.1 9200
> Trying 127.0.0.1...
> Connected to 127.0.0.1.
> Escape character is '^]'.
> '^]'.
> Connection closed by foreign host.

If I disable the elasticsearch.ssl.verify to false in kibana.yml , ELK works fine.

What could be the issue over here.

Regards,
Vinod