Filebeat SSL logs reporting 'tls: failed to send closeNotify alert'

Elastic Stack Beats
1

I have managed to implement an SSL connection between a FileBeat (8.13) agent and a LogStash (8.13) but have noticed in the FileBeat syslog logs that the SSL connections to the LogStash server are getting reset and these entries are repeated after a few entries are shipped to the LogStash server;

<30>1 2024-06-12T08:55:00.082579-07:00 svsandd2 systemd 1 - - Starting system activity accounting tool...
<30>1 2024-06-12T08:55:00.110172-07:00 svsandd2 systemd 1 - - sysstat-collect.service: Succeeded.
<30>1 2024-06-12T08:55:00.110305-07:00 svsandd2 systemd 1 - - Started system activity accounting tool.
<131>1 2024-06-12T08:55:12.617865-07:00 svsandd2 filebeat 13788 - - 2024-06-12T08:55:12.617-0700 ERROR [logstash] map[file.line:100 file.name:logstash/sync.go function:github.com/elastic/beats/v7/libbeat/outputs/logstash.(*syncClient).reconnect] error closing connection to logstash host <redacted>:7514: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp <redacted>:54892->*<redacted>*:7514: write: connection reset by peer, reconnecting...#011{"ecs.version": "1.6.0"}
<131>1 2024-06-12T08:55:23.859990-07:00 svsandd2 filebeat 13788 - - 2024-06-12T08:55:23.859-0700 ERROR [logstash] map[file.line:100 file.name:logstash/sync.go function:github.com/elastic/beats/v7/libbeat/outputs/logstash.(*syncClient).reconnect] error closing connection to logstash host <redacted>:7514: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp <redacted>:42916->*<redacted>*:7514: write: connection reset by peer, reconnecting...#011{"ecs.version": "1.6.0"}

Nevertheless, the logs are being shipped as expected, so the actual SSL handling is working.

Herewith the relevant configuration snippets;

/etc/filebeat/filebeat.yml

output.logstash:
    hosts:
      - "*<redacted>*:7514"
      - "*<redacted>*:7514"
    loadbalance: true
    ttl: 30
    pipelining: 0
    ssl:
        enabled: true
        key: /etc/pki/tls/private/filebeat.key
        certificate: /etc/pki/tls/certs/filebeat.pem
        certificate_authorities: [ /etc/pki/ca-trust/source/anchors/*<redacted>*.CA_crt ]
        verification_mode: strict

/etc/logstash/conf.d/pipeline.conf

input {
    # listen for FileBeat log objects on the internal network interface
    beats {
        host => "127.0.0.1"
        port => 7512
    }
    beats {
        port => 7514
        ssl_enabled => true
        ssl_key => "/etc/pki/tls/private/logstash.key"
        ssl_certificate => "/etc/pki/tls/certs/logstash.pem"
        ssl_certificate_authorities => [ "/etc/pki/ca-trust/source/anchors/*<redacted>*.CA_crt" ]
        ssl_client_authentication => required
    }
}

output {
    # dump entry to /var/log/logstash/pipeline.output
    file {
        path => "/var/log/logstash/pipeline.output"
    }

    # discard output to /dev/null
    # null {}
}

I am uncertain what is happening here and whether it is a configuration or an implementation issue but it is disconcerting as it is happening frequently. Any suggestions on how fix the issue to stop these disconcerting messages would be much appreciated.

2

Hi @wcraven , we are facing the same error in our environment, did you able to find the reason for this error generation? please let us know on the same.