Attempted to resurrect connection to dead ES instance, but got an error... Received fatal alert: bad_certificate"}

Elastic Stack Logstash
1

Hello,

I wanted to run the old logstash config that once was working.

It is not working anymore I guess that because now Elasticsearch is set up with

xpack.security.transport.ssl.verification_mode: certificate

Logstah output config

output {
# file {
#   path => "/etc/logstash/conf.d/tests/snmp.txt"
# }
 stdout { codec => rubydebug }
  elasticsearch {
        hosts => ["https://fqdn.local:9200"]
        index => "network-devices-%{+YYYY.MM.dd}"
        user => "${es_log}"
        password => "${es_pwd}"
        cacert => "/path/elastic-ca.crt"
        ssl=> true
      }
}

Error log

]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/hostname/SNMP-CPU-hostname.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2022-01-18T16:21:40,445][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-01-18T16:21:40,455][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-01-18T16:21:40,790][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-01-18T16:21:41,969][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-01-18T16:21:42,932][INFO ][org.reflections.Reflections] Reflections took 66 ms to scan 1 urls, producing 119 keys and 417 values
[2022-01-18T16:21:45,059][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://fqdn.local:9200"]}
[2022-01-18T16:21:45,466][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/]}}
[2022-01-18T16:21:45,955][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/][Manticore::ClientProtocolException] Received fatal alert: bad_certificate"}
[2022-01-18T16:21:46,084][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/etc/logstash/conf.d/hostname/SNMP-CPU-hostname.conf"], :thread=>"#<Thread:0x3984f460 run>"}
[2022-01-18T16:21:47,113][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.03}
[2022-01-18T16:21:47,158][INFO ][logstash.inputs.snmp     ][main] using plugin provided MIB path /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/mibs/logstash
[2022-01-18T16:21:47,181][INFO ][logstash.inputs.snmp     ][main] using plugin provided MIB path /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/mibs/ietf
[2022-01-18T16:21:49,485][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-01-18T16:21:49,538][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
      "system.memory.used.norm" => 0.53,
     "system.memory.free.bytes" => 853880248,
                  "ip.observer" => "10.10.10.10",
     "system.memory.used.bytes" => 976386924,
                   "@timestamp" => 2022-01-18T15:21:49.589Z,
            "system.cpu.norm.1" => 0.36,
                      "host.ip" => "10.10.10.10",
      "system.memory.free.norm" => 0.47,
                "host.hostname" => "hostname",
                         "tags" => [
        [0] "snmp",
        [1] "metrics"
    ],
    "system.memory.total.bytes" => 1830267172
}
[2022-01-18T16:21:51,162][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/][Manticore::ClientProtocolException] Received fatal alert: bad_certificate"}
[2022-01-18T16:21:56,348][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/][Manticore::ClientProtocolException] Received fatal alert: bad_certificate"}
[2022-01-18T16:22:01,516][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/][Manticore::ClientProtocolException] Received fatal alert: bad_certificate"}
[2022-01-18T16:22:06,675][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/][Manticore::ClientProtocolException] Received fatal alert: bad_certificate"}
[2022-01-18T16:22:11,831][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/][Manticore::ClientProtocolException] Received fatal alert: bad_certificate"}
[2022-01-18T16:22:17,015][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://%251B%255BA%251Blogstash_writter:xxxxxx@fqdn.local:9200/][Manticore::ClientProtocolException] Received fatal alert: bad_certificate"}

ELK version: 7.16-2
Logstash version: 7.16-3 (the message was the same on 7.16-2)

How can I make it work again?

2

How can I replicate the configuration that I can provide for example like for the beats?

  hosts: ["https://fqdn:9200"]
  username: "${ES_LOG}"
  password: "${ES_PWD}"
  ssl.certificate_authorities: ["/path/elastic-ca.crt"]
  ssl.certificate: "/path/beats.crt"
  ssl.key: "/path/beats.key"
  ssl.key_passphrase: "${KEY_PWD}"
3

After reading the docs 10 times I noticed the solution.

  hosts: ["https://fqdn:9200"]
  username: "${es_log}"
  password: "${es_pwd}"
  cacert => "/path/elastic-ca.crt"

  keystore => "/path/logstash.p12"
  keystore_password => "${key_pwd}"

  ilm_enabled => false
  manage_template => false
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.