SSL/TLS on ELK cluster

jaymode · July 12, 2016, 11:45am

Ok let's take a few steps back and make sure that openssl can read the certificate file.

Can you run the following:

openssl x509 -in cert.crt -noout -text
openssl version

Also, if you open the crt file in a text editor, can you verify that the first line is -----BEGIN CERTIFICATE----- and the last line is -----END CERTIFICATE-----

vienodp · July 12, 2016, 12:18pm

PFA is the output of commands.
If I open cert.pem then yes I do have the same lines mentioned by you.
If I open cert.crt then it does not have those.

jaymode · July 12, 2016, 12:36pm

Does openssl x509 -in cert.pem -noout -text work?

vienodp · July 12, 2016, 12:51pm

Here is the output -

[root@irldxvm022 vinodar3]# openssl x509 -in cert.pem -noout -text
unable to load certificate
139660169418568:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1343:
139660169418568:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:393:Type=X509_CINF
139660169418568:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:777:Field=cert_info, Type=X509
139660169418568:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:pem_oth.c:83:
[root@irldxvm022 vinodar3]#

jaymode · July 12, 2016, 1:32pm

Ok this doesn't actually seem like a pem file at all.

Next commands to try:

openssl pkcs7 -print_certs -in cert.pem -out cert_pem.cer
openssl x509 -in cert._pem.cer -noout -text

vienodp · July 13, 2016, 7:21am

Here is the output

jaymode · July 13, 2016, 11:48am

Try this:

Create a new file single_cert.cer
Open cert_pem.cer and copy starting at the first -----BEGIN CERTIFICATE----- line to the first -----END CERTIFICATE----- line.
Paste that into single_cert.cer
openssl x509 -in single_cert.cer -noout -text
openssl pkcs12 -export -inkey irldxvm022.key -in single_cert.cer -name irldxvm022 -out irldxvm022.p12
1. NOTE: Make sure you use the matching key for the certificate; I simply based the values on what was previously provided

Also, is there a reason you generated the CSR with openssl and did not use the instructions in the documentation?

vienodp · July 13, 2016, 12:22pm

Hi Jay,

Thanks, the steps you have given worked. Please let me know the next step.

Regards,
Vinod

jaymode · July 13, 2016, 12:36pm

I believe these two commands should work:

After that check keytool -list -v irldxvm022.jks and ensure you have a privateKeyEntry. You may also need to import the intermediate and root CA certificates into the keystore.

vienodp · July 14, 2016, 9:49am

Hi Jay,
Thanks yes both these commands too worked. I could also import root and intermediate certificates. Now I have 3 aliases in keystore. One for private key and 2 for root and intermediate certificates.
I did same on both nodes and configured keystore.
Now how to crosscheck if SSL/TLS encryption between ELK node is working.

Kibana to ES and Logstash to ES configuration for SSL/TLS is also done.

jaymode · July 14, 2016, 11:37am

You can enable DEBUG logging and you should see log messages like "SSL handshake completed for channel". Or you can use something like openssl s_client to open a connect and see that the socket is using SSL:

openssl s_client -showcerts -connect localhost:9300

vienodp · July 15, 2016, 3:37am

Ok thanks.

I have enabled Shield plugin and configured the SSL/TLS encryption. Now my kibana is not starting up.

I can see these entries in logstash logs -

timestamp=>"2016-07-15T13:01:51.995000+0530", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["https://9.126.112.72:9200/\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused", :class=>"Manticore::SocketException", :client_config=>{:hosts=>["https://9.126.112.72:9200/"], :ssl=>{:enabled=>true, :ca_file=>"/etc/logstash/cert.pem"}, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{:enabled=>true, :ca_file=>"/etc/logstash/cert.pem"}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :headers=>{"Authorization"=>"Basic dmlub2RhcjNAaW4uaWJtLmNvbTp0aWdlUkAzMjE="}, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :level=>:error

and these entries in elastcisearch logs

[2016-07-15 14:07:25,129][WARN ][shield.transport.netty ] [irldxvm022] received plaintext http traffic on a https channel, closing connection [id: 0x3d9f3207, /9.126.112.35:45330 => /9.126.112.72:9200]

jaymode · July 15, 2016, 10:19am

I don't think those messages are related. Is the 9.126.112.35 IP of the logstash instance? "Connection refused" indicates a failed connection attempt on the socket and nothing to do with ssl

vienodp · July 15, 2016, 11:12am

I have 2 nodes in a cluster , 35 and 72 with ELK/Shield installed on it. Yes, those messages may not be related. But what could be the reason of connection refused where ES is working fine and listening on the required ports. I also checked with this command which gives me status as green.
Firewall and selinux parameters also checked and those are disabled.

curl -XGET -k -u vinodar3@in.ibm.com -p 'https://9.126.112.72:9200/_cluster/health?pretty=true'

[root@irldxvm022 ~]# netstat -tulpn | grep 9200
tcp        0      0 ::ffff:9.126.112.72:9200    :::*                        LISTEN      3573/java
tcp        0      0 fe80::250:56ff:fea0:77:9200 :::*                        LISTEN      3573/java
tcp        0      0 ::ffff:127.0.0.1:9200       :::*                        LISTEN      3573/java
tcp        0      0 ::1:9200                    :::*                        LISTEN      3573/java
[root@irldxvm022 ~]# netstat -tulpn | grep 9300
tcp        0      0 ::ffff:9.126.112.72:9300    :::*                        LISTEN      3573/java
tcp        0      0 fe80::250:56ff:fea0:77:9300 :::*                        LISTEN      3573/java
tcp        0      0 ::ffff:127.0.0.1:9300       :::*                        LISTEN      3573/java
tcp        0      0 ::1:9300                    :::*                        LISTEN      3573/java

vienodp · July 15, 2016, 11:24am

Here is my output file -

output {
  elasticsearch {
    user => "vinodar3@in.ibm.com"
    password => "xyz@123"
    ssl => true
    cacert => "/etc/logstash/cert.pem"
    hosts => ["https://9.126.112.72:9200"]
    manage_template => false
    document_type => "%{[@metadata][type]}"
  }
}

jaymode · July 15, 2016, 11:36am

I am not sure what could cause that. I think you should open a new topic in the #logstash section as things appear to be working fine on the Shield side.

vienodp · July 15, 2016, 11:46am

Ok, I will do that. But after enabling Shield plugin these things started. Kibana is not coming up and not showing anything in the logs. I have gone through the "Using Logstash with Shield" and "Using Kibana with Shield" and configured the things accordingly still no luck.

https://discuss.elastic.co/t/logstash-es-communication-issue-and-kibana-not-coming-up/55613

jaymode · July 15, 2016, 12:09pm

What is your kibana configuration? Most likely Kibana is still trying to use plaintext. The logstash aspect is different which is why I asked you to start a new topic

vienodp · July 15, 2016, 12:18pm

For Kibaana I am using https in ES url elasticsearch.url: "https://9.126.112.72:9200" and configured the ssl.crt and ssl.key and ssl.ca certificates.
ES user name and password are LDAP user and its password.

jaymode · July 15, 2016, 12:45pm

did you configure elasticsearch.ssl.ca?

Topic		Replies	Views
LDAP integrated ELK with Shield Elasticsearch elastic-stack-security	26	5202	July 6, 2017
ELK Stack + TLS + Headache Kibana elastic-stack-security	15	2123	September 17, 2021
Securing ELK Elasticsearch	2	306	November 20, 2020
ELK cluster with x-pack kibana is not working with https Kibana	4	802	June 12, 2017
SSL Error in Kibana logs Kibana	15	1804	October 14, 2020

SSL/TLS on ELK cluster

Related topics