Configuring x-pack for SSl communication betweeen elasticsearch and logstash

vinod_hy · June 12, 2017, 7:57am

Please help me in using x-pack.

My Requirement:
My setup is kibana and elasticsearch resides on one machine. Logstash and filebeat resides on another machine.
My requirement is to secure the connection with SSL certificates between logstash and elasticsearch.

Steps i followed:

(1) I have installed x-pack on ElasticSearch.Using default username/password (elastic/changeme).

(2) I have self signed certificates generated for client and server. In elasticsearch.yml, i made the following changes,

xpack.ssl.key: "D:/Softwares/ELK/elasticsearch-5.4.0/elasticsearch-5.4.0/config/x-pack/ELK_SSL_Certificates/server/pkcs8.key"
xpack.ssl.certificate: "D:/Softwares/ELK/elasticsearch-5.4.0/elasticsearch-5.4.0/config/x-pack/ELK_SSL_Certificates/server/cert.pem"
xpack.ssl.certificate_authorities: [ "D:/Softwares/ELK/elasticsearch-5.4.0/elasticsearch-5.4.0/config/x-pack/ELK_SSL_Certificates/testca/cacert.pem" ]

(3) I am able to successfully establish SSL connection established between filebeat and logstash, Below is filebeat.yml and logstash.conf changes.

filebeat.yml:

output.logstash:
  #The Logstash hosts
    hosts: ["localhost:5043"]
    ssl.enabled: true
    ssl.certificate_authorities: ["D:/Softwares/ELK/ELK_SSL_Certificates/testca/cacert.pem"]
    ssl.certificate: "D:/Softwares/ELK/ELK_SSL_Certificates/client/cert.pem"
    ssl.key: "D:/Softwares/ELK/ELK_SSL_Certificates/client/pkcs8.key"

logstash.conf:

input {
    beats {
        port => "5043"
		ssl => true
		ssl_certificate_authorities => "D:/Softwares/ELK/ELK_SSL_Certificates/testca/cacert.pem"
		ssl_certificate => "D:/Softwares/ELK/ELK_SSL_Certificates/server/cert.pem"
		ssl_key => "D:/Softwares/ELK/ELK_SSL_Certificates/server/pkcs8.key"
		ssl_verify_mode => "force_peer"
    }
}
filter{
	grok
	{
		match => {"message" =>"%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:keyval}"}
	}
	kv 
	{
		  source => "keyval"
		  field_split => ","
		  #trimkey => "\s"
		  remove_field => [ "keyval" ]
    }
}

output {
    elasticsearch {
	hosts => ["localhost:9200"]
	user => "elastic"
	password => "changeme"
	}
	stdout { codec => rubydebug }
}

(4) now i want to secure the connection with SSL between elasticsearch and logstash in the same way i was able to secure connection between filebeat and logstash.
I have not installed x-pack in logstash yet. Please let me know on how to achieve it. Am i on right track?

vinod_hy · June 13, 2017, 4:35pm

can anyone please help me

joshbressers · June 13, 2017, 4:55pm

This link to the x-pack documentation should be what you're looking for
https://www.elastic.co/guide/en/x-pack/current/logstash.html#ls-http-ssl

vinod_hy · June 14, 2017, 5:53am

Hi @joshbressers ,
Thanks for the link. I added the below lines in logstash.config file,
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "elastic"
password => "changeme"
ssl => true
cacert => ["D://Softwares//ELK//ELK_SSL_Certificates//testca//cacert.pem"]
}
stdout { codec => rubydebug }
}

after adding ssl lines as mentioned in the link provided, i get below error when i start logstash ,
11:21:49.029 [[main]-pipeline-manager] ERROR logstash.agent - Pipeline aborted due to error {:exception=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>

Please tell me is there any thing else i am missing here

vinod_hy · June 14, 2017, 6:06am

@magnusbaeck .. Please give me your inputs here.

TimV · June 14, 2017, 6:50am

Please post the error message in full, and include in a "code block" in your message by putting 3 backticks before and after the message like this:
```
11:21:49.029 [[main]-pipeline-manager] ERROR logstash.agent - Pipeline aborted due to error {:exception=>#
```

vinod_hy · June 14, 2017, 6:57am

@TimV

'''ERROR logstash.agent - Pipeline aborted due to error {:exception=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?'''

The above is the error i get.

TimV · June 14, 2017, 7:01am

It doesn't look like you have enabled SSL inside elasticsearch.

You need to explicitly turn on SSL for the http (REST) interface.

vinod_hy · June 14, 2017, 9:20am

@TimV,

Below are the changes i have made in my elasticsearch.yml,
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.client_authentication: required
xpack.ssl.key: "D:/Softwares/ELK/elasticsearch-5.4.0/elasticsearch-5.4.0/config/x-pack/ELK_SSL_Certificates/server/pkcs8.key"
xpack.ssl.certificate: "D:/Softwares/ELK/elasticsearch-5.4.0/elasticsearch-5.4.0/config/x-pack/ELK_SSL_Certificates/server/cert.pem"
xpack.ssl.certificate_authorities: "D:/Softwares/ELK/elasticsearch-5.4.0/elasticsearch-5.4.0/config/x-pack/ELK_SSL_Certificates/testca/cacert.pem"

Am i missing any thing else here.

vinod_hy · June 14, 2017, 9:57am

@TimV

Hello Tim,

I am able to establish https connection to elasticsearch. In browser, i could see this

But in elasticsearch.log, i get ,

'''io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:'''

TimV · June 14, 2017, 11:41am

That means that something tried to connect to your ES cluster using HTTP rather than HTTPS. Without more information, I can't tell you where that connection came from.

vinod_hy · June 14, 2017, 12:09pm

@TimV

Hello Tim,

The exception is no more seen. I tried accessing to elasticsearch using http a few times. Thats the reason it threw that exception. I did not realise. Apologise for the same.

Now the next issue is i am unable to connect to elasticsearch from logstash and kibana. below is my logstash.conf output section,
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "elastic"
password => "changeme"
ssl => true
cacert => ["D://Softwares//ELK//ELK_SSL_Certificates//testca//cacert.pem"]
}
stdout { codec => rubydebug }
}

I get below error,
''':error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}'''

vinod_hy · June 15, 2017, 3:36am

@TimV

Hi Tim,

My elasticsearch is also up and running with credentials elastic/changeme.

TimV · June 15, 2017, 4:01am

This indicates that you have a problem with the CA chain for your ES certificate.
My best guess is this:

I'm not a logstash expert, but I don't believe that the cacert option to the logstash-elasticsearch-output plugin allows arrays.

Can you try:

cacert => "D:/Softwares/ELK/ELK_SSL_Certificates/testca/cacert.pem"

vinod_hy · June 15, 2017, 4:15am

@TimV
i tried that also.. Still same issue.

For your information, i have installed x-pack in elasticsearch and made changes in elasticsearch.yml as shared above. I have also installed x-pack in logstash and had made below changes,

In logstash.yml,
xpack.monitoring.elasticsearch.url: ["https://132.186.102.39:9200"]
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme

Now i commented those lines also. The logstash.yml is a fresh file now with all lines commented.

Please note that i have added below line in elasticsearch.yml,
network.host: 132.186.102.39

now i see logstash trying to connect to elasticsearch using logstash_system and elastic as username. Please find the logs below,

'''10:39:24.433 [Ruby-0-Thread-7: D:/Softwares/ELK/logstash-5.4.0/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:234] INFO
logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
10:39:24.437 [Ruby-0-Thread-10: D:/Softwares/ELK/logstash-5.4.0/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:532] DEBUG logstash.pipeline - Pushing flush onto pipeline
10:39:24.454 [Ruby-0-Thread-6: D:/Softwares/ELK/logstash-5.4.0/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:234] WARN
logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x50b697b9 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system
_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:
9200/][Manticore::SocketException] Connection refused: connect"}'''

'''10:39:24.612 [Ruby-0-Thread-12: D:/Softwares/ELK/logstash-5.4.0/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:234] INF
O logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@132.186.102.39:9200/, :path=>"/"}
10:39:24.647 [Ruby-0-Thread-12: D:/Softwares/ELK/logstash-5.4.0/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:234] WAR
N logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTPS:0x27d5513c URL:https://elastic:xxxxxx@132.186.102.39:9200/>, :error_type=>LogStash:
:Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@132.186.102.39:9200/][Manticore::ClientProtocolException] KeyUsage does not allow digit
al signatures"}'''

Strange thing is that when it uses logstash_system, it uses localhost instead of ip adress. When it uses elastic , it uses ip_address instead of localhost.

vinod_hy · June 15, 2017, 6:04am

Can anyone from logstash team help me here .

system · July 13, 2017, 6:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.