ERR Connecting Error

Hey Guys

I setup an ELK Server on my server, inside a VM, the ELK's servers IP is 192.168.154.168. I set a client to talk to the ELK Server, it's ip is 192.168.154.155, it's using filebeat over port 5443 to talk to the ELK Server.

When I run systemctl start filebeat and then tail -f the log file I see:

2017-08-18T12:25:12-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.logstash.publish.write_bytes=151 libbeat.publisher.published_events=2046
2017-08-18T12:25:12-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49234->192.168.154.168:5443: i/o timeout
2017-08-18T12:25:42-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:25:43-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49236->192.168.154.168:5443: i/o timeout
2017-08-18T12:26:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:26:15-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49238->192.168.154.168:5443: i/o timeout
2017-08-18T12:26:42-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:26:49-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49240->192.168.154.168:5443: i/o timeout
2017-08-18T12:27:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:27:27-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49242->192.168.154.168:5443: i/o timeout

When I run tcpdump port 5443 on the ELK Server I get:

root@elk:~# tcpdump port 5443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
12:33:45.995959 IP 192.168.154.155.49256 > elk-master.5443: Flags [F.], seq 2122303693, ack 924170643, win 229, options [nop,nop,TS val 2372580921 ecr 16683389], length 0
12:33:46.032278 IP elk-master.5443 > 192.168.154.155.49256: Flags [.], ack 1, win 235, options [nop,nop,TS val 16690899 ecr 2372580921], length 0

My IPTABLES rules are:

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A PREROUTING -i eth0 -p tcp -m tcp --dport 9300 -j DNAT --to-destination 192.168.154.168
-A PREROUTING -i br0 -p tcp -m tcp --dport 5443 -j DNAT --to-destination 192.168.154.168
-A POSTROUTING -d 192.168.154.168/32 -p tcp -m tcp --dport 9300 -j SNAT --to-source 192.168.154.155
-A POSTROUTING -d 192.168.154.168/32 -p tcp -m tcp --dport 5443 -j SNAT --to-source 192.168.154.155

When I watch the IPTABLES for PKT's I get

72 3744 SNAT tcp -- any any anywhere elk-master tcp dpt:9300 to:192.168.154.155
12 720 SNAT tcp -- any any anywhere elk-master tcp dpt:5443 to:192.168.154.155

Which I'll also see if I do a tcpdump on the client.

What I don't understand is what is causing the error, here is the output section of my filebeat.yml conffiguration:

94 output.logstash:
95 # The Logstash hosts
96 hosts: ["elk-master:5443"]
97 bulk_max_size: 5443
98 ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]
99 template.name: "filebeat"
100 template.path: "filebeat.template.json"
101 template.overwrite: false

I've seen some other posts about this on the forum, but they didn't help, anyone have ideas what to try?

Thanks
Docmur

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.