This is a basic task that could easily be done with tcpdump and I was hoping to do it with packetbeat as well.
Let's say a process on the current instance is trying to connect to an external endpoint on port 443. This being HTTPS, packetbeat.protocols.http would not work.
But suppose the remote endpoint is down anyway. My client on the current instance tries to connect, sends out a SYN packet, but it gets an RST (reset) packet. The whole exchange takes less than a second.
If I go on Kibana and search for this flow by dest.ip, it does get recorded, but it's broken in several ways.
Packetbeat does not seem to realize that the transaction is done. The RST packet was sent back in under 1 second, but packetbeat keeps following the "flow" until it hits the expiration time. This is wrong. An RST packet means the transaction is done, stop following it.
There is no way to distinguish between sessions that connect successfully (SYN, SYN/ACK, ACK, data exchange, FIN, FIN/ACK, FIN, FIN/ACK), and sessions that fail (are terminated with RST, or with an ICMP port unreachable or something like that).
This greatly reduces the usefulness of packetbeat when troubleshooting connection failures. It seems like packetbeat was written with the higher layers in mind, but at lower layers it is not quite usable. What I'm saying is, a network engineer would find packetbeat lacking in a number of ways, some of which are listed above. It's great when things work well, but it's less useful when there are failures below application level.
What would be VERY useful is something akin to Follow TCP Stream in Wireshark. If that is not possible, at least indicate somehow the TCP flows that failed to connect, and make sure to catch the RST packets, and ICMP packets, that terminate a flow early. And indicate on the flow the specific way in which it has failed (e.g. "RST received").