Hello,
I am using the Elastic Stack 6 on 3 servers running Debian 9.1.
- One ("server2") has Elasticsearch, Logstash, Kibana installed
- All three have Filebeat and Metricbeat installed to collect data
Logstash is configured to accept beats input on port 5044 on a local IP on LAN.
input {
beats {
port => 5044
ssl => false
}
}
The filebeat and metricbeat instances on all 3 servers manage to send data to logstash, which filters it ans stores it in ElasticSearch.
output {
if [@metadata][beat] == 'filebeat' {
if [fileset][module] {
elasticsearch {
hosts => "server2:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[fileset][module]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
} else if "postfix" in [tags] {
elasticsearch {
hosts => "server2:9200"
manage_template => false
index => "%{[@metadata][beat]}-postfix-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
} else {
elasticsearch {
hosts => "server2:9200"
manage_template => false
index => "%{[@metadata][beat]}-nonmodule-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
}
if [@metadata][beat] == 'metricbeat' {
elasticsearch {
hosts => "server2:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[metricset][module]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
}
The data is visible in Kibana, so that seems to be OK.
We have a problem however with connections from beats, because a large number of them seem to stay as "ESTABLISHED" and after a while, they cause a "Too many open files" error (after reaching more than 4700 connections last night).
The problem seems to happen only on connections from the other 2 servers (server1 and server3), not from server2 itself.
Less than 2,5 hours after restarting logstash, we see this ESTABLISHED connection count on port 5044 :
- from server1 : 525 connections
- from server2 : 2 connections
- from server3 : 264 connections (we stopped filebeat on this server to have only metricbeat running, so that might explain why there are half as many connection as from server1).
Expected behaviour would be to have connections closed cleanly or reused, but not have new connections opening all the time without closing.
When I check connections on server1 and server3, I see only one that is ESTABLISHED on server3 (for metricbeat) and 2 on server1 (for both beats), as well as some connections that are "FIN_WAIT1" (10 on server3 and 20 on server1).