Filebeat won't talk to logstash after upgrade to 6.x

Elastic Stack Beats
1

Upgraded the system to the latest 6.x, but when filebeat tries to talk to logstash I get

2018-05-30T15:17:14.438Z ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 172.16.1.241:38472->172.16.2.22:5044: write: connection reset by peer
2018-05-30T15:17:15.438Z ERROR pipeline/output.go:92 Failed to publish events: write tcp 172.16.1.241:38472->172.16.2.22:5044: write: connection reset by peer

in the filebeat log and nothing in the logstash log.

To check network connectivity I try

root@et3-tim-1:/var/log/filebeat# nc 172.16.2.22 5044
agarbage
root@et3-tim-1:/var/log/filebeat#

from the machine running filebeat, and this time it does get through, as witnessed by logstash getting upset with the garbage:

[2018-05-30T16:17:52,399][INFO ][org.logstash.beats.BeatsHandler] [local: 172.16.2.22:5044, remote: 172.16.1.241:38790] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 97
[2018-05-30T16:17:52,399][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 97

Which doesn't make sense to me - I usually expect "connection reset by peer" to be a networking level problem, but if I'm not using filebeat I can connect from the same source host to the same port on the same destination using nc. What have I failed to notice?

2

What was the prior version? I've had problems when more than one type even when my indices only had one type. The template had a type that was never used.

3

Previous version, which was all working fine a couple of hours earlier, was 5.6.something.

4

Looking at the network traffic, I do rather get the impression that "connection reset by peer" is misleading, as it doesn't appear to mean, as one would expect, that an RST packet has been received in response to the SYN.

So far it looks like a TCP connection is set up, and some data goes back and forth, and the last thing that happens is that Logstash sends the binary packet 32:41:00:00:00:01 to Filebeat, and that's the end of the conversation (no FIN packets or RST packets seem to be sent either way).

Where do I find a specification of the protocol, and/or (preferably) a Wireshark dissector?

5

OK, it looks like the

messages are completely spurious and misleading, and the reason that no Filebeat output was appearing in Elasticsearch is that I hadn't got the changes to _type etc right in the Logstash configuration, so Logstash was silently discarding all my documents. The only log messages I could find at the right time were those ERROR messages in the Filebeat logs, so naturally I started investigating there.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.