Filebeat connections stuck in CLOSE_WAIT

Elastic Stack Logstash
1

I have multiple servers running filebeat sending logs to two logstash servers (all in AWS), after a couple of days the logstash servers stop ingesting and report "Too many files open". An lsof shows thousands of connections from the filebeat servers stuck in CLOSE_WAIT.

Filebeat config:

filebeat.prospectors:
- input_type: log

  paths: "/var/log/syslog"
  fields:
    type: syslog
  fields_under_root: true

- input_type: log

  paths:
    - /var/log/applog.log
  fields:
    type: applog
  fields_under_root: true
  json.keys_under_root: true
  json.overwrite_keys: true

output.logstash:
  hosts:
      - logstash-01:5044
      - logstash-02:5044
  loadbalance: true
  ssl.certificate_authorities:
    - /etc/ssl/logstash-ca.crt

Logstash config:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    client_inactivity_timeout => 300
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    date {
      target => "syslog_timestamp"
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
     elasticsearch {
         ssl => true 
         hosts => ["es1:9200","es2:9200","es3:9200"]
         index =>'%{type}-%{+YYYY.MM.dd}'
         user => 'username'
         password => '**********'
         ssl_certificate_verification => false
      }
}

Any help appreciated!

Edit: Filebeat 5.6.5, Logstash 5.6.4 all on Ubuntu

2

@dwjvaughan This issue should be fixed if you upgrade your logstash-input-beats plugin.

Thanks,

Rob

3

Thanks for the update, have updated the plugin - fingers crossed!

4

We have same issue on Centos 7.4 logstash version 5.6.6 and the beats plugin is update, so i think the issue is not resolved

Step to reproduce

  • install 2 elastich data node
  • install on other machine logstash that point to the 2 data server
  • shutdown one of the data node
    now on logstash machine u can see connection in CLOSED_WAIT that point to data server

In this state healthcheck not working too and the only way to restore full functionality is restart logstash service

5

Hi @Luxiter

Which version of the beats plugin are you using?

Also, can you post a copy of your configuration, redacting any sensitive information.

Thanks,

Rob

6

plugin version for logstash-input-beats (5.0.6)
We have done some test, try to use hostname IP or a FQDN but the issue is the same if elastic data node go down logstash remain in closed wait connection.

Version rpm 5.6.6-1 (logstash-5.6.6-1.noarch)
Operating System Centos 7.4 x64
Config logstash.yml
# Settings file in YAML
#

path.data: /var/lib/logstash

path.config: /etc/logstash/conf.d

....
....

config 30-elasticsearch-output.conf (conf.d folder)
output {
elasticsearch {
hosts => ["http://192.168.1.191:9200", "http://ansible3:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
config 01-beats-input.conf
input {
beats {
port => 5044
}
}

7

@Luxiter

Thanks.

Is your behavior the same as stated above, where you will eventually run out of file descriptors due to thousands of connections in a CLOSE_WAIT state? Does Logstash recover once the elastic data node recovers?

Rob

8

The only way for restore full functionality is restart logstash service.
The CLOSE_WAIT connection is only one for data node (one to one)

In our test we have 2 elastic data server if we stop one data node, logstash remain in close wait on http port for data node that we have shutdown, if we stop other elastic data server, logstash remain in close wait on second data node too, for restore connection we have to restart logstash

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.