Hey Guys
I setup an ELK Server on my server, inside a VM, the ELK's servers IP is 192.168.154.168. I set a client to talk to the ELK Server, it's ip is 192.168.154.155, it's using filebeat over port 5443 to talk to the ELK Server.
When I run systemctl start filebeat and then tail -f the log file I see:
2017-08-18T12:25:12-04:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.logstash.publish.write_bytes=151 libbeat.publisher.published_events=2046
2017-08-18T12:25:12-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49234->192.168.154.168:5443: i/o timeout
2017-08-18T12:25:42-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:25:43-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49236->192.168.154.168:5443: i/o timeout
2017-08-18T12:26:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:26:15-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49238->192.168.154.168:5443: i/o timeout
2017-08-18T12:26:42-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:26:49-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49240->192.168.154.168:5443: i/o timeout
2017-08-18T12:27:12-04:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=151
2017-08-18T12:27:27-04:00 ERR Connecting error publishing events (retrying): read tcp 192.168.154.155:49242->192.168.154.168:5443: i/o timeout
When I run tcpdump port 5443 on the ELK Server I get:
root@elk:~# tcpdump port 5443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
12:33:45.995959 IP 192.168.154.155.49256 > elk-master.5443: Flags [F.], seq 2122303693, ack 924170643, win 229, options [nop,nop,TS val 2372580921 ecr 16683389], length 0
12:33:46.032278 IP elk-master.5443 > 192.168.154.155.49256: Flags [.], ack 1, win 235, options [nop,nop,TS val 16690899 ecr 2372580921], length 0
My IPTABLES rules are:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A PREROUTING -i eth0 -p tcp -m tcp --dport 9300 -j DNAT --to-destination 192.168.154.168
-A PREROUTING -i br0 -p tcp -m tcp --dport 5443 -j DNAT --to-destination 192.168.154.168
-A POSTROUTING -d 192.168.154.168/32 -p tcp -m tcp --dport 9300 -j SNAT --to-source 192.168.154.155
-A POSTROUTING -d 192.168.154.168/32 -p tcp -m tcp --dport 5443 -j SNAT --to-source 192.168.154.155
When I watch the IPTABLES for PKT's I get
72 3744 SNAT tcp -- any any anywhere elk-master tcp dpt:9300 to:192.168.154.155
12 720 SNAT tcp -- any any anywhere elk-master tcp dpt:5443 to:192.168.154.155
Which I'll also see if I do a tcpdump on the client.
What I don't understand is what is causing the error, here is the output section of my filebeat.yml conffiguration:
94 output.logstash:
95 # The Logstash hosts
96 hosts: ["elk-master:5443"]
97 bulk_max_size: 5443
98 ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]
99 template.name: "filebeat"
100 template.path: "filebeat.template.json"
101 template.overwrite: false
I've seen some other posts about this on the forum, but they didn't help, anyone have ideas what to try?
Thanks
Docmur