Upgraded the system to the latest 6.x, but when filebeat tries to talk to logstash I get
2018-05-30T15:17:14.438Z ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 172.16.1.241:38472->172.16.2.22:5044: write: connection reset by peer
2018-05-30T15:17:15.438Z ERROR pipeline/output.go:92 Failed to publish events: write tcp 172.16.1.241:38472->172.16.2.22:5044: write: connection reset by peer
in the filebeat log and nothing in the logstash log.
To check network connectivity I try
root@et3-tim-1:/var/log/filebeat# nc 172.16.2.22 5044
from the machine running filebeat, and this time it does get through, as witnessed by logstash getting upset with the garbage:
[2018-05-30T16:17:52,399][INFO ][org.logstash.beats.BeatsHandler] [local: 172.16.2.22:5044, remote: 172.16.1.241:38790] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 97
[2018-05-30T16:17:52,399][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 97
Which doesn't make sense to me - I usually expect "connection reset by peer" to be a networking level problem, but if I'm not using filebeat I can connect from the same source host to the same port on the same destination using nc. What have I failed to notice?