SSL/TLS on ELK cluster

Ok thanks.

I have enabled Shield plugin and configured the SSL/TLS encryption. Now my kibana is not starting up.

I can see these entries in logstash logs -

timestamp=>"2016-07-15T13:01:51.995000+0530", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["https://9.126.112.72:9200/"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused", :class=>"Manticore::SocketException", :client_config=>{:hosts=>["https://9.126.112.72:9200/"], :ssl=>{:enabled=>true, :ca_file=>"/etc/logstash/cert.pem"}, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{:enabled=>true, :ca_file=>"/etc/logstash/cert.pem"}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :headers=>{"Authorization"=>"Basic dmlub2RhcjNAaW4uaWJtLmNvbTp0aWdlUkAzMjE="}, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :level=>:error

and these entries in elastcisearch logs

[2016-07-15 14:07:25,129][WARN ][shield.transport.netty ] [irldxvm022] received plaintext http traffic on a https channel, closing connection [id: 0x3d9f3207, /9.126.112.35:45330 => /9.126.112.72:9200]

I don't think those messages are related. Is the 9.126.112.35 IP of the logstash instance? "Connection refused" indicates a failed connection attempt on the socket and nothing to do with ssl

I have 2 nodes in a cluster , 35 and 72 with ELK/Shield installed on it. Yes, those messages may not be related. But what could be the reason of connection refused where ES is working fine and listening on the required ports. I also checked with this command which gives me status as green.
Firewall and selinux parameters also checked and those are disabled.

curl -XGET -k -u vinodar3@in.ibm.com -p 'https://9.126.112.72:9200/_cluster/health?pretty=true'

[root@irldxvm022 ~]# netstat -tulpn | grep 9200
tcp        0      0 ::ffff:9.126.112.72:9200    :::*                        LISTEN      3573/java
tcp        0      0 fe80::250:56ff:fea0:77:9200 :::*                        LISTEN      3573/java
tcp        0      0 ::ffff:127.0.0.1:9200       :::*                        LISTEN      3573/java
tcp        0      0 ::1:9200                    :::*                        LISTEN      3573/java
[root@irldxvm022 ~]# netstat -tulpn | grep 9300
tcp        0      0 ::ffff:9.126.112.72:9300    :::*                        LISTEN      3573/java
tcp        0      0 fe80::250:56ff:fea0:77:9300 :::*                        LISTEN      3573/java
tcp        0      0 ::ffff:127.0.0.1:9300       :::*                        LISTEN      3573/java
tcp        0      0 ::1:9300                    :::*                        LISTEN      3573/java

Here is my output file -

output {
  elasticsearch {
    user => "vinodar3@in.ibm.com"
    password => "xyz@123"
    ssl => true
    cacert => "/etc/logstash/cert.pem"
    hosts => ["https://9.126.112.72:9200"]
    manage_template => false
    document_type => "%{[@metadata][type]}"
  }
}

I am not sure what could cause that. I think you should open a new topic in the #logstash section as things appear to be working fine on the Shield side.

Ok, I will do that. But after enabling Shield plugin these things started. Kibana is not coming up and not showing anything in the logs. I have gone through the "Using Logstash with Shield" and "Using Kibana with Shield" and configured the things accordingly still no luck.

https://discuss.elastic.co/t/logstash-es-communication-issue-and-kibana-not-coming-up/55613

What is your kibana configuration? Most likely Kibana is still trying to use plaintext. The logstash aspect is different which is why I asked you to start a new topic

For Kibaana I am using https in ES url elasticsearch.url: "https://9.126.112.72:9200" and configured the ssl.crt and ssl.key and ssl.ca certificates.
ES user name and password are LDAP user and its password.

did you configure elasticsearch.ssl.ca?

Yes that is already configured in kibana.yml
elasticsearch.ssl.ca: /etc/elasticsearch/shield/cert.pem

What cert is that? Is it the intermediate CA cert?

No it is not intermediate cert.
Using following we generated cert.pem and I am using this cert-

Here are the steps we followed to generate the certificate -
1.Put the request using -

openssl req -nodes -newkey rsa:2048 -sha256 -keyout myserver.key -out server.csr
2.On the CA website we got the certificates available in following formats -

DER, CRT, PKCS7b

( When I download PKCS7b file , it takes .pem extension )

So it is the cert of the elasticsearch server? It should be the intermediate certificate that you use for elasticsearch.ssl.ca and you may have also include the root CA certificate as well like:

elasticsearch.ssl.ca: [ "/path/to/root.pem", "/path/to/intermediate.pem" ]

Please make sure these are in PEM format. You may need to use the openssl commands we used before if they are not

Yes it was cert of ES server. Ok , I have int. cert and root cert in der format which I converted to pem format using -
e.g.
openssl x509 -inform der -in caintermediatecert.der -out caintermediatecert.pem

Did for both int and root certs and configured as given by you. Kibana service is still not starting.

The kiabana log entries are 1 day old after that its not generating any logs -

{"type":"log","@timestamp":"2016-07-15T05:35:24+00:00","tags":["warning","elasticsearch"],"pid":1960,"message":"No living connections"}
{"type":"log","@timestamp":"2016-07-15T05:35:27+00:00","tags":["warning","elasticsearch"],"pid":1960,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2016-07-15T05:35:27+00:00","tags":["warning","elasticsearch"],"pid":1960,"message":"No living connections"}

So you get not output at all from running "bin/kibana"?

I am running kibana using /etc/init.d/kibana start/stop

If I start using /bin/kibana , I get following -

[root@irldxvm022 kibana]# bin/kibana serve
FATAL [Error: error:0906D06C:PEM routines:PEM_read_bio:no start line]

This implies that one of your certificate/key files isn't really a PEM file or has extra text in it outside of the ----BEGIN and -----END lines. I suggest you inspect each file

Ok I checked both files but I do not see any spaces or extra characters as such. I converted those from der format to pem and used as it is.

We have few days left of Shield trial license.
To speed up the resolution , should I set up webex session. Is that fine with you ? Are you working in EST time zone ?

Thanks,
Vinod

Hi Jay,

Now we trying with the commands given in the official document-
We ran following commands -

keytool -genkey -alias elk01 -keystore elk01.jks -keyalg RSA -keysize 2048 -validity 712 -ext san=dns:irldxvm022.irl.in.ibm.com,ip:9.126.112.72

keytool -certreq -alias elk01 -keystore elk01.jks -file elk01.csr -keyalg rsa -ext san=dns:irldxvm022.irl.in.ibm.com,ip:9.126.112.72

After that we are uploading the csr file on the portal for signing , however we are getting the errors.
PFA is the screenshot of errors.

Hi Vinod,

I am not sure what causes that. The keysize you specified has the proper minimum size. Did you use the same name when running keytool ("CN=9.126.112.72,OU=Research,L=New Delhi,ST=N/A,C=IN") ?

I think the portal administrators may be able to help you better as I am not familiar with it.

-Jay