ERR SSL client failed to connect (1.x to 5.x upgrade)

Elastic Stack Beats
1

I am running into this error. After what appears to be events successfully published to output.

2017/06/15 17:43:43.708785 publish.go:88: INFO Start sending events to output
2017/06/15 17:43:43.708851 log.go:115: INFO Harvester started for file: /usr/local/bin/ELK/iperf.10.129.0.30.out

After the data is published I am getting this error.

2017/06/15 17:43:46.227461 output.go:87: DBG output worker: publish 40 events
2017/06/15 17:43:46.231619 transport.go:125: ERR SSL client failed to connect with: read tcp 10.129.0.33:40610->10.XX.XX.XX:1337: read: connection reset by peer
2017/06/15 17:43:46.231665 single.go:126: INFO Connecting error publishing events (retrying): read tcp 10.129.0.33:40610->10.XX.XX.XX:1337: read: connection reset by peer
2017/06/15 17:43:46.231690 single.go:154: INFO send fail
2017/06/15 17:43:46.231712 single.go:161: INFO backoff retry: 1s

I disabled the firewall on both sides to verify that was not the cause.

2

Hi @jm_peterson,

Could you please provide more details about your setup? I am particularly interested in your filebeat.yml settings

3 
############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      # Paths that should be crawled and fetched. Glob based paths.
      # To fetch all ".log" files from a specific level of subdirectories
      # /var/log/*/*.log can be used.
      # For each file found under this path, a harvester is started.
      # Make sure not file is defined twice as this can lead to unexpected behaviour.
      paths:
        - /usr/local/bin/ELK/*.out
        #- /var/log/secure
        #- /var/log/messages
        #- /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*

      # Configure the file encoding for reading files with international characters
      # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
      # Some sample encodings:
      #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
      #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
      encoding: plain

      # Type of the files. Based on this the way the file is read is decided.
      # The different types cannot be mixed in one prospector
      #
      # Possible options are:
      # * log: Reads every line of the log file (default)
      # * stdin: Reads the standard in
      input_type: log

      # Set to true to store the additional fields as top level fields instead
      # of under the "fields" sub-dictionary. In case of name conflicts with the
      # fields added by Filebeat itself, the custom fields overwrite the default
      # fields.
      fields_under_root: false

      # Type to be published in the 'type' field. For Elasticsearch output,
      # the type defines the document type these entries should be stored
      # in. Default: log
      document_type: syslog

  # Name of the registry file. Per default it is put in the current working
  # directory. In case the working directory is changed after when running
  # filebeat again, indexing starts from the beginning again.
  registry_file: /var/lib/filebeat/registry

  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["10.XXX.X.XX:1337"]

    # Number of workers per Logstash host.
    #worker: 1

    # The maximum number of events to bulk into a single batch window. The
    # default is 2048.
    bulk_max_size: 1024 

    # Set gzip compression level.
    #compression_level: 3

    # Optional load balance the events between the Logstash hosts
    #loadbalance: true

    # Optional index name. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat
    # top topbeat and for Filebeat to filebeat.
    #index: filebeat

    # Optional TLS. By default is off.
    tls:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

############################# Logging #########################################

# There are three options for the log ouput: syslog, file, stderr.
# Under Windos systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

  # Send all logging output to syslog. On Windows default is false, otherwise
  # default is true.
  #to_syslog: true

  # Write all logging output to files. Beats automatically rotate files if rotateeverybytes
  # limit is reached.
  #to_files: false

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    #path: /var/log/mybeat

    # The name of the files where the logs are written to.
    #name: mybeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    #keepfiles: 7

  # Enable debug output for selected components. To enable all selectors use ["*"]
  # Other available selectors are beat, publish, service
  # Multiple selectors can be chained.
  #selectors: [ ]

  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  level: info
4

What version of Filebeat are you running? In 5.x the TLS settings were renamed to be ssl. See Breaking Changes in 5.0.

The indentation in your config looks a little off, but maybe that was a copy/paste issue.

5

@andrewkroh

I believe the indentation error is copy/paste, I made sure to verify that.
My Filebeat was on 1.x, and have since upgraded them to 5.x

Updated portion of filebeat.yml

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.XXX.X.XX:1337"]

  # Optional TLS. By default is off.
  # List of root certificates for HTTPS server verifications
  #tls.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for TLS client authentication
  #tls.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #tls.certificate_key: "/etc/pki/client/cert.key"

  # Optional SSL configuration options. SSL is off by default.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

It appears filebeat is forwarding the information now, and logstash is having another issue. Being new to the forum I want to make sure to follow guidelines, so would I open that as another post?

6

Yeah, if it's a separate unrelated issue use a new topic.

7

Thank you for the help. It turned out to be an issue with the certificate generation/updating the the correct filebeat version.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.