# Filebeat connections stuck in CLOSE\_WAIT
**URL:** https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747
**Category:** Logstash
**Created:** [January 2, 2018, 9:29am UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747 "2018-01-02T09:29:02Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@dwjvaughan](https://discuss.elastic.co/u/dwjvaughan)
#### Post date: [January 2, 2018, 9:29am UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/1 "2018-01-02T09:29:03Z")
I have multiple servers running filebeat sending logs to two logstash servers (all in AWS), after a couple of days the logstash servers stop ingesting and report "Too many files open". An lsof shows thousands of connections from the filebeat servers stuck in CLOSE\_WAIT.
Filebeat config:
```
filebeat.prospectors:
- input_type: log
paths: "/var/log/syslog"
fields:
type: syslog
fields_under_root: true
- input_type: log
paths:
- /var/log/applog.log
fields:
type: applog
fields_under_root: true
json.keys_under_root: true
json.overwrite_keys: true
output.logstash:
hosts:
- logstash-01:5044
- logstash-02:5044
loadbalance: true
ssl.certificate_authorities:
- /etc/ssl/logstash-ca.crt
```
Logstash config:
```
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
client_inactivity_timeout => 300
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
}
date {
target => "syslog_timestamp"
match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
}
}
output {
elasticsearch {
ssl => true
hosts => ["es1:9200","es2:9200","es3:9200"]
index =>'%{type}-%{+YYYY.MM.dd}'
user => 'username'
password => ' **********'
ssl_certificate_verification => false
}
}
```
Any help appreciated!
Edit: Filebeat 5.6.5, Logstash 5.6.4 all on Ubuntu
---
### Author:  [@RobBavey](https://discuss.elastic.co/u/RobBavey)
#### Post date: [January 5, 2018, 8:05pm UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/2 "2018-01-05T20:05:55Z")
@dwjvaughan This issue should be fixed if you upgrade your `logstash-input-beats` plugin.
Thanks,
Rob
---
### Author:  [@dwjvaughan](https://discuss.elastic.co/u/dwjvaughan)
#### Post date: [January 8, 2018, 9:40am UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/3 "2018-01-08T09:40:00Z")
Thanks for the update, have updated the plugin - fingers crossed!
---
### Author:  [@Luxiter](https://discuss.elastic.co/u/Luxiter)
#### Post date: [January 19, 2018, 11:58pm UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/4 "2018-01-19T23:58:19Z")
We have same issue on Centos 7.4 logstash version 5.6.6 and the beats plugin is update, so i think the issue is not resolved
Step to reproduce
- install 2 elastich data node
- install on other machine logstash that point to the 2 data server
- shutdown one of the data node
now on logstash machine u can see connection in CLOSED\_WAIT that point to data server
In this state healthcheck not working too and the only way to restore full functionality is restart logstash service
---
### Author:  [@RobBavey](https://discuss.elastic.co/u/RobBavey)
#### Post date: [January 22, 2018, 4:53pm UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/5 "2018-01-22T16:53:00Z")
Hi @Luxiter
Which version of the beats plugin are you using?
Also, can you post a copy of your configuration, redacting any sensitive information.
Thanks,
Rob
---
### Author:  [@Luxiter](https://discuss.elastic.co/u/Luxiter)
#### Post date: [January 22, 2018, 5:42pm UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/6 "2018-01-22T17:42:52Z")
plugin version for logstash-input-beats (5.0.6)
We have done some test, try to use hostname IP or a FQDN but the issue is the same if elastic data node go down logstash remain in closed wait connection.
Version rpm 5.6.6-1 (logstash-5.6.6-1.noarch)
Operating System Centos 7.4 x64
Config logstash.yml
# Settings file in YAML
#
```
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
```
....
....
config 30-elasticsearch-output.conf (conf.d folder)
output {
elasticsearch {
hosts =\> ["[http://192.168.1.191:9200](http://192.168.1.191:9200)", "[http://ansible3:9200](http://ansible3:9200)"]
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document\_type =\> "%{[@metadata][type]}"
}
}
config 01-beats-input.conf
input {
beats {
port =\> 5044
}
}
---
### Author:  [@RobBavey](https://discuss.elastic.co/u/RobBavey)
#### Post date: [January 22, 2018, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/7 "2018-01-22T18:53:41Z")
@Luxiter
Thanks.
Is your behavior the same as stated above, where you will eventually run out of file descriptors due to thousands of connections in a CLOSE\_WAIT state? Does Logstash recover once the elastic data node recovers?
Rob
---
### Author:  [@Luxiter](https://discuss.elastic.co/u/Luxiter)
#### Post date: [January 24, 2018, 8:14am UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/8 "2018-01-24T08:14:35Z")
The only way for restore full functionality is restart logstash service.
The CLOSE\_WAIT connection is only one for data node (one to one)
In our test we have 2 elastic data server if we stop one data node, logstash remain in close wait on http port for data node that we have shutdown, if we stop other elastic data server, logstash remain in close wait on second data node too, for restore connection we have to restart logstash
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 21, 2018, 8:14am UTC](https://discuss.elastic.co/t/filebeat-connections-stuck-in-close-wait/113747/9 "2018-02-21T08:14:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.