I'm trying to get a basic ELK setup, but I've been having difficulties verifying that Filebeat publishes to Logstash. Filebeat reports that it can't publish events because "use of closed network connection".
I suspect it's a network issue between client and server, but I'd like to gather ideas for what to do next. After some Googling, it looks like the phrase "use of a closed network connection" is a message that golang uses.
Packet captures suggest that the server is responding to the client. I have a sneaking suspicion that the issue is actually Logstash and not Filebeat, because the error messages suggests that the server isn't responding properly. But I'm posting it here because the immediate issue is visible in Filebeat.
Filebeat error
#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# Client using filebeat: Filebeat failed to publish
# Host: 10.10.10.81
# Command: cat /etc/filebeat/filebeat.yml
2018-02-11T14:10:15.372-0800 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp 10.10.10.81:52152->10.10.10.93:5044: i/o timeout
2018-02-11T14:10:15.372-0800 ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 10.10.10.81:52152->10.10.10.93:5044: use of closed network connection
2018-02-11T14:10:16.373-0800 ERROR pipeline/output.go:92 Failed to publish events: write tcp 10.10.10.81:52152->10.10.10.93:5044: use of closed network connection
Client info
#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# Client info
# Host: 10.10.10.81
# Command: miscellaneous info
$ filebeat version
filebeat version 6.2.1 (amd64), libbeat 6.2.1
$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)
Client has an active TCP connection to server
#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# Client using filebeat: client (http-29) has a connection to logstash server (logs-1)
# Host: 10.10.10.81
# Command: lsof -i -P # I filtered for only the relevant row
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 2331 root 5u IPv4 393900 0t0 TCP http-29.node.dc1.app.local:52062->logs-1.node.dc1.app.local:5044 (ESTABLISHED)
Filebeat conf in the client
#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
# Client using filebeat: filebeat configuration
# Host: 10.10.10.81
# Command: cat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/messages
output.logstash:
hosts: ["logstash.service.app.local:5044"]
Server packet capture
#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# Logstash server: packet capture shows that the server is receiving and responding to
# Host: 10.10.10.93
# Command: tcpdump -i eth0 -nn host 10.10.10.81 and port 5044
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
14:38:46.178866 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [S], seq 1666666507, win 29200, options [mss 1460,sackOK,TS val 51722092 ecr 0,nop,wscale 6], length 0
14:38:46.178980 IP 10.10.10.93.5044 > 10.10.10.81.52618: Flags [S.], seq 1518647658, ack 1666666508, win 28960, options [mss 1460,sackOK,TS val 3216965 ecr 51722092,nop,wscale 7], length 0
14:38:46.179312 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], ack 1, win 457, options [nop,nop,TS val 51722093 ecr 3216965], length 0
14:38:46.257717 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 1:1449, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257720 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 1449:2897, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257721 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 2897:4345, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257723 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 4345:5793, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257725 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 5793:7241, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257729 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 7241:8689, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257731 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 8689:10137, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257733 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 10137:11585, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257735 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 11585:13033, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.257737 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 13033:14481, ack 1, win 457, options [nop,nop,TS val 51722171 ecr 3216965], length 1448
14:38:46.267452 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 14481:15929, ack 1, win 457, options [nop,nop,TS val 51722181 ecr 3216965], length 1448
14:38:46.471405 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 1:1449, ack 1, win 457, options [nop,nop,TS val 51722385 ecr 3216965], length 1448
14:38:46.880479 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], seq 1:1449, ack 1, win 457, options [nop,nop,TS val 51722794 ecr 3216965], length 1448
14:38:47.500773 IP 10.10.10.93.5044 > 10.10.10.81.52618: Flags [S.], seq 1518647658, ack 1666666508, win 28960, options [mss 1460,sackOK,TS val 3218287 ecr 51722794,nop,wscale 7], length 0
14:38:47.501165 IP 10.10.10.81.52618 > 10.10.10.93.5044: Flags [.], ack 1, win 457, options [nop,nop,TS val 51723414 ecr 3216965], length 0
Server logstash configuration
#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# Logstash server: logstash configuration
# Host: 10.10.10.93
# Command: cat /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
Server profile to listen to filebeat
#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
# Logstash server: listen to filebeat
# Host: 10.10.10.93
# Command: cat /etc/logstash/conf.d/default_port_5044.conf
input {
beats {
host => "logstash.service.app.local"
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch.service.app.local:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}