SSL/TLS on ELK cluster

Elastic Stack Elasticsearch
21

Ok let's take a few steps back and make sure that openssl can read the certificate file.

Can you run the following:

  1. openssl x509 -in cert.crt -noout -text
  2. openssl version

Also, if you open the crt file in a text editor, can you verify that the first line is -----BEGIN CERTIFICATE----- and the last line is -----END CERTIFICATE-----

22

PFA is the output of commands.
If I open cert.pem then yes I do have the same lines mentioned by you.
If I open cert.crt then it does not have those.

screen2.JPG955×296 46.2 KB

23

Does openssl x509 -in cert.pem -noout -text work?

24

Here is the output -

[root@irldxvm022 vinodar3]# openssl x509 -in cert.pem -noout -text
unable to load certificate
139660169418568:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1343:
139660169418568:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:393:Type=X509_CINF
139660169418568:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:777:Field=cert_info, Type=X509
139660169418568:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:pem_oth.c:83:
[root@irldxvm022 vinodar3]#

25

Ok this doesn't actually seem like a pem file at all.

Next commands to try:

  1. openssl pkcs7 -print_certs -in cert.pem -out cert_pem.cer
  2. openssl x509 -in cert._pem.cer -noout -text
26

Here is the output

2.JPG1043×727 117 KB
1.JPG966×727 185 KB

27

Try this:

  1. Create a new file single_cert.cer
  2. Open cert_pem.cer and copy starting at the first -----BEGIN CERTIFICATE----- line to the first -----END CERTIFICATE----- line.
  3. Paste that into single_cert.cer
  4. openssl x509 -in single_cert.cer -noout -text
  5. openssl pkcs12 -export -inkey irldxvm022.key -in single_cert.cer -name irldxvm022 -out irldxvm022.p12
    1. NOTE: Make sure you use the matching key for the certificate; I simply based the values on what was previously provided

Also, is there a reason you generated the CSR with openssl and did not use the instructions in the documentation?

28

Hi Jay,

Thanks, the steps you have given worked. Please let me know the next step.

Regards,
Vinod

29

I believe these two commands should work:

After that check keytool -list -v irldxvm022.jks and ensure you have a privateKeyEntry. You may also need to import the intermediate and root CA certificates into the keystore.

30

Hi Jay,
Thanks yes both these commands too worked. I could also import root and intermediate certificates. Now I have 3 aliases in keystore. One for private key and 2 for root and intermediate certificates.
I did same on both nodes and configured keystore.
Now how to crosscheck if SSL/TLS encryption between ELK node is working.

Kibana to ES and Logstash to ES configuration for SSL/TLS is also done.

31

You can enable DEBUG logging and you should see log messages like "SSL handshake completed for channel". Or you can use something like openssl s_client to open a connect and see that the socket is using SSL:

openssl s_client -showcerts -connect localhost:9300
32

Ok thanks.

I have enabled Shield plugin and configured the SSL/TLS encryption. Now my kibana is not starting up.

I can see these entries in logstash logs -

timestamp=>"2016-07-15T13:01:51.995000+0530", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["https://9.126.112.72:9200/\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused", :class=>"Manticore::SocketException", :client_config=>{:hosts=>["https://9.126.112.72:9200/"], :ssl=>{:enabled=>true, :ca_file=>"/etc/logstash/cert.pem"}, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{:enabled=>true, :ca_file=>"/etc/logstash/cert.pem"}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :headers=>{"Authorization"=>"Basic dmlub2RhcjNAaW4uaWJtLmNvbTp0aWdlUkAzMjE="}, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :level=>:error

and these entries in elastcisearch logs

[2016-07-15 14:07:25,129][WARN ][shield.transport.netty ] [irldxvm022] received plaintext http traffic on a https channel, closing connection [id: 0x3d9f3207, /9.126.112.35:45330 => /9.126.112.72:9200]

33

I don't think those messages are related. Is the 9.126.112.35 IP of the logstash instance? "Connection refused" indicates a failed connection attempt on the socket and nothing to do with ssl

34

I have 2 nodes in a cluster , 35 and 72 with ELK/Shield installed on it. Yes, those messages may not be related. But what could be the reason of connection refused where ES is working fine and listening on the required ports. I also checked with this command which gives me status as green.
Firewall and selinux parameters also checked and those are disabled.

curl -XGET -k -u vinodar3@in.ibm.com -p 'https://9.126.112.72:9200/_cluster/health?pretty=true'

[root@irldxvm022 ~]# netstat -tulpn | grep 9200
tcp        0      0 ::ffff:9.126.112.72:9200    :::*                        LISTEN      3573/java
tcp        0      0 fe80::250:56ff:fea0:77:9200 :::*                        LISTEN      3573/java
tcp        0      0 ::ffff:127.0.0.1:9200       :::*                        LISTEN      3573/java
tcp        0      0 ::1:9200                    :::*                        LISTEN      3573/java
[root@irldxvm022 ~]# netstat -tulpn | grep 9300
tcp        0      0 ::ffff:9.126.112.72:9300    :::*                        LISTEN      3573/java
tcp        0      0 fe80::250:56ff:fea0:77:9300 :::*                        LISTEN      3573/java
tcp        0      0 ::ffff:127.0.0.1:9300       :::*                        LISTEN      3573/java
tcp        0      0 ::1:9300                    :::*                        LISTEN      3573/java
35

Here is my output file -

output {
  elasticsearch {
    user => "vinodar3@in.ibm.com"
    password => "xyz@123"
    ssl => true
    cacert => "/etc/logstash/cert.pem"
    hosts => ["https://9.126.112.72:9200"]
    manage_template => false
    document_type => "%{[@metadata][type]}"
  }
}
36

I am not sure what could cause that. I think you should open a new topic in the #logstash section as things appear to be working fine on the Shield side.

37

Ok, I will do that. But after enabling Shield plugin these things started. Kibana is not coming up and not showing anything in the logs. I have gone through the "Using Logstash with Shield" and "Using Kibana with Shield" and configured the things accordingly still no luck.

https://discuss.elastic.co/t/logstash-es-communication-issue-and-kibana-not-coming-up/55613

38

What is your kibana configuration? Most likely Kibana is still trying to use plaintext. The logstash aspect is different which is why I asked you to start a new topic

39

For Kibaana I am using https in ES url elasticsearch.url: "https://9.126.112.72:9200" and configured the ssl.crt and ssl.key and ssl.ca certificates.
ES user name and password are LDAP user and its password.

40

did you configure elasticsearch.ssl.ca?