Logstash not able to connect secured (ssl) Elastic search cluster

Elastic Stack Logstash
1

I have installed Logstash,elasticsearch and kibana in single instance and installed X-pack also for TLS communication. enabled ssl communication in elasticsearch and kibana working good but logstash unable to connect elasticsearch , but i can curl elasticsearch url https://localhost:9200 there is no firewall blocking also, I have generated open ssl certificate and key file and kept in elasticsearch

    output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => elastic
    password => password
    manage_template => false
    ssl_certificate_verification => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

  }
}

elasticsearch config file

cluster.name: my-application
network.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /opt/elasticsearch/ca/ca.key
xpack.security.http.ssl.certificate: /opt/elasticsearch/ca/ca.crt

[2018-05-16T05:28:16,421][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:17,201][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2018-05-16T05:28:21,422][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-16T05:28:21,422][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-16T05:28:21,424][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:21,425][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:22,202][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2018-05-16T05:28:26,425][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-16T05:28:26,426][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-16T05:28:26,427][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:26,427][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2018-05-16T05:28:27,201][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
root@5c417caecc5f:/var/log/logstash#
2

Please make the effort to format your post to be as readable as possible - there's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of incorrectly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

3

This seems a bit weird - why are you using a CA key/cert for your HTTP interface?

Can you please provide copy-and-paste the curl command and output. It doesn't make any sense for curl to be able to connect but logstash can't

4

Problem is logstash is pointing to http://localhost even though I have written HTTPS. you can see that in logs .

5

Sorry, I didn't pay enough attention to the error message.
The error in your logs is due to logstash monitoring - you need to configure monitoring separately to your elasticsearch output.

6

I didn't enable monitoring in logstash , i don't want also , my problem is i want to enable tls communication between logstash and elasticsearch can you help me to do that

7

If you installed X-Pack in Logstash then you implicitly enabled X-Pack monitoring.
You can disable it if you want, see:

There's no sign that you have a problem enabling TLS between logstash and elasticsearch. You have shown a series of warning messages related to monitoring, but I can't see anything that suggests you have a TLS problem.

8

but it worked good until i setup TLS communication, after configuring SSL keys i am getting problem

9

Can you describe the problem?

The problem you posted above is entirely to do with X-Pack monitoring. You can solve it by configuring monitoring to use SSL, or by turning monitoring off.

But that problem does not affect your normal Elasticsearch output, so if you have a problem getting you data from Logstash into Elasticsearch, then you will need to provide details about that error.

10

i have elk stack setup 6.2.3, all logs are coming from file beats to logstash . recently i have installed x pack on logstash,elasticsearch & kibana for TLS encryption purpose , first i installed xpack and configured user authentication between all services. it worked good for me, now i need to configure TLS encryption over https between all services logstash, elasticsearch and kibana. i generated ssl keys with openssl command and placed in logstash, elasticsearch and kibana , kibana is able to connect elastic search and working well but logstash is unable to connect to elasticsearch . this is what my problem .......... and you said that disable x.pack monitroing , how to do that?

11

Hi @ashok9177,

As you mentioned that TLS setup with ES - Kibana is working fine but not ES - Logstash,

logstash is unable to connect to elasticsearch

Could you upload logs from logstash showing failing connections to elasticsearch for what you have mentioned above. That would help us go through the exception stack trace to know what is the cause?

For disabling x-pack monitoring @TimV as already given you directions pointing to documentation. Reposting the link:

Look for xpack.monitoring.enabled setting to disable monitoring which is enabled by default.

Hope this helps.

Regards,
Yogesh Gaikwad

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.