I am trying a POC for my organisation and have the following setup :
- Filebeat on Linux Server
- Logstash, ElasticSearch and Kibana on Windows Server 2012
A simple flow of picking up log files from Filebeat to Kibana worked really well. To make it more secure, I had installed x-pack on Kibana, ES and Logstash. After some initial hurdles I was able to make the same connection from Filebeat to Kibana work.
To make things more compatible for Production Environment, I decided to try SSL communication between Filebeat to Logstash.
I've tried a lot of methods and everyone of them failed. Therefore, after searching for hours I found a step-by-step guide (https://github.com/Busindre/How-to-configure-SSL-for-FileBeat-and-Logstash-step-by-step) and I followed it religiously to obtain all certificates (etc.) from openssl (version 1.0)
Note : I've generated all the certificates on Linux server where filebeat was configured. The reason I've done this is because I couldn't generate certificates on Windows Server as I am not sure how it works.
After I was done with generating certificates on Linux Server, I did a file transfer to my local system and then to Windows Server.
Now, I've started my Logstash with following config
input {
beats {
port => 5044
client_inactivity_timeout => 3600
ssl => true
ssl_certificate_authorities => ["C:/ELKStack/Final_Cert/ca.crt"]
ssl_certificate => "C:/ELKStack/Final_Cert/server.crt"
ssl_key => "C:/ELKStack/Final_Cert/server.key"
ssl_verify_mode => "none"
}
}
This worked fine. Now I went back to filebeat which had following config :
output.logstash:
# The Logstash hosts
hosts: ["cloud_ip:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["path/ca.crt"]
# Certificate for SSL client authentication
ssl.certificate: "path/client.crt"
# Client Certificate Key
ssl.key: "path/client.key"
ssl.supported_protocols: "TLSv1.2"
As I start the filebeat server I get following results :
INFO Harvester started for file: /path*.log
2017/11/13 15:21:18.157222 sync.go:85: ERR Failed to publish events caused by: read tcp XX->XX:5044: read: connection reset by peer
2017/11/13 15:21:18.157263 single.go:91: INFO Error publishing events (retrying): read tcp XX->XX:5044: read: connection reset by peer
At the same time I got this in my logstash cmd :
[2017-11-13T10:21:01,087][INFO ][logstash.inputs.metrics ] Monitoring License OK
[2017-11-13T10:21:18,050][INFO ][org.logstash.beats.BeatsHandler] Exception: not an SSL/TLS record: 32570000000a3243000006b0785ed458cd72dbc81
18692cde6e729ba50b92d4401fc359162d5ca20b44b47fc114065d7ebb85423a009ce0a9ca10703598cac27c943e494f7ca35954b6a06044548b2b5f4aaecf2452a764ffff7a0
bf9efa0bc330f60c63efe76bf3db054a1213494cf7da94ab259aae99f2c4b4cc7324d274cd194df11c89346f2cf35b49179849b2589aae59b79dcebee3ec3b8da9d372eb8eeb7
46aed7ae727d332339e8b084dd73cb824e2802fe541c45984421ea43cc90e72220f2893399107110a7946d98cd70aa37c36cb509a6eb3d168b6da8e557589b2652ecf2a5e2e30
cb48a2bc761a30e297a0dc02a7eeb61a6ebd6bd92d1b06a3a331c03e7828a447a239ba4099e49072129fb21845baa22c0927df4de702b3394f6300b38cffda6464a1d4cfe2884
649c4594a6c27372d73ce33f93ede258a8c7266ba66abd6ae35cc9b1b9df3df18c65eb263ce1f0db0f95080c862787f7ce61347677de2ce6837da8f754691f0df1ac6d7f91326
bcd5b05a1d6793f0bfe7b6dd884e44f15fff45a7e3384ec3766cdbaedbdd6d4edfb76ddbfba1f32268dac5d90023a49718c349ce2506f826c74c825bf9e98205431ac729be250
2c7e73f632415cd4b293239cc92411f7a00168498a99ed33fbdc369380ec182a3c18f0f090f8b5b73abed7b24310aa5f739269485525096404fc9d79ab53a58f09cc7ab636489
9c430f9c4e53b99525d3d512a10781b6cf62141e5f2cab2e4c8948506ee8dffbd39ffce3cdf9303fd7c72322339e154a437c33ca17d08366bbbb3e485932a50b654add6f95e17
dc75177bcd5a8b5da6db060cc9ee39ca4b3f16c63aa63b79d0ae79eb15ba1631e114939d3ce6c7c9c0a42d3222fde1ca38b50bbe5349441951095b0b2765ab25a7db060c4034c
89c4385ca988943fe16a71ce53e8c17561e746c5a869e1ec4a913501a35c50b9d25aafb75b477107fd90e72252f9b86e6e1f5fd7e35ad186446a1543cee4fc251251ea2e197db
22a49935c8e854752edd67edd69769acf1aede633ad5b0a7a8113410b7bfbf5da86ed77d581f1521e4a29e8792eb5478ae671260589e4304f255da614b5f5fbb21ecf979c05a4
90bccf0fd779f0afa23961c946ff20cbf242a7b2e6b388c71817c463642aac3b51548e946197dafb9845256d7d709bf5418d0fe99808bcf4529ee14417f47e58db17beac99a32
2990a1253966c5f665dcc90c69bd8c72246712275f19c965db395e0512e4394b2bfcea4228d45bcdd0f5b27ea65b0a52a4db8efa67fb5a402d7374ffb311584652492db342f17
0259a4fdf14f0365fab6e94f33147d9c51867169f40b1f3fad6ea7f5cbc6cf5786b1f7bf1dc7cf9360a7b6d37ef6988b96f92168a127dd2db428e6d8c74f3a502320cf245fa86
e48c94af54f809734c3185c1871f0d65c
[2017-11-13T10:21:19,332][INFO ][org.logstash.beats.BeatsHandler] Exception: not an SSL/TLS record: [Deleted]
I am really confused as to how to go ahead right now. It will really helpful if you could help. Thanks
