Logstash unable to connect to Elasticsearch over https

[Bastian_Jager (Bastian Jäger)]

Hello,

I am facing connection issues from logstash to my elastic stack. The very strange this is, that it worked until yesterday. Today (without any modification from my side) I get:

Dec 17 22:00:55 ubuntu logstash-app[25173]: [2021-12-17T22:00:55,819][WARN ]
  [logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, 
  but got an error. {:url=>"https://user:xxxxxx@elastic.xxx.com:9200/", :error_type=>
  LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>
  "Elasticsearch Unreachable: [https://user:xxxxxx@elastic.xxx.com:9200/]
  [Manticore::ClientProtocolException] PKIX path validation failed:
  java.security.cert.CertPathValidatorException: validity check failed"}

My logstash output section is:

	output {
	  elasticsearch {
		hosts => "https://elastic.xxx.com:9200"
		ssl => true
		ssl_certificate_verification => false # add this to test
		document_id => "..."
		index => "..."
		user => "user"
		password => "password"
		doc_as_upsert => true
		action => "update"
	  }
	}

I did not need / have certification check for ther cert and would like to leave it like this. I have no keystore or truststore. Only "simple SSL".

This works from the cmd line:

curl -X GET https://user:password@elastic.xxx.com:9200 -k

I did a lot of research. I saw this Issues with SSL certificate validation in logstash output http plugin · Issue #21 · logstash-plugins/logstash-output-http · GitHub
But so far nothing helped.

Anyone have an idea what I can do?

[Badger]

@Bastian_Jager

ssl_certificate_verification => false

According to this issue, setting :ssl_verify to false does not turn off as much verification as 'curl -k'. If the problem really did start without any changes being made I would look for an expiration of something in the certification path.

[Bastian_Jager (Bastian Jäger)]

Thanks for the reply.
But so what would be the solution to connect to elastic without a cert?

[Bastian_Jager (Bastian Jäger)]

Now our 2nd server instance of logstash (other host, other logstash version) is having the issue.

Can this be somewhat related to the Log4j security breach? The only thing we did was running apt update && apt upgrade on the host of our elastic server. But elastic is running in a docker even.

[Badger]

@Bastian_Jager

But so what would be the solution to connect to elastic without a cert?

The issue I linked to suggests you cannot completely disable certificate validation.

[TimV (Tim Vernum)]

Do you mean running Elasticsearch without a server certificate? We do not support that.

[Bastian_Jager (Bastian Jäger)]

No, not exactly.

I am running elastic from docker with

      - xpack.license.self_generated.type=basic 
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/my-cert.key
      - xpack.security.http.ssl.certificate=$CERTS_DIR/my-cert.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/my-cert.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/my-cert.key

and e.g. my kibana (same host, also via docker) connects without certificate validation like this

      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: ${KIBANA_SYSTEM_USER}
      ELASTICSEARCH_PASSWORD: ${KIBANA_SYSTEM_PW}
      ELASTICSEARCH_SSL_VERIFICATIONMODE: none
      XPACK_REPORTING_ENCRYPTIONKEY: my-password
      XPACK_SECURITY_ENCRYPTIONKEY: my-password
      XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: SomeEncryptionKey1234567890

Until some days ago I was able to connect to the elastic instance (with traefik proxy in between) with a logstash (again docker)

version: '3.7'

services:
    sensor-data-logstash:
        network_mode: "host"
        container_name: sensor-data-logstash
        image: logstash:7.16.1
        env_file:
          - .env
        volumes:
          - ./log:/home/user/log
          - ./config:/usr/share/logstash/config 

and the config config/logstash.yml

http.host: 0.0.0.0
path.config: /usr/share/logstash/config/logstash.config

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "https://my-elastic.com:9200" ]
xpack.monitoring.elasticsearch.username: ${LOGSTASH_USER}
xpack.monitoring.elasticsearch.password: ${LOGSTASH_PW}
xpack.monitoring.elasticsearch.ssl.verification_mode: none

and the logstash.config output section from the original post.

This is not working anymore.