Can packetbeat send pcap to http end point?

Elastic Stack Beats
1

My use case in short:

I want to run packetbeat on one of my servers and pipe the pcap directly into snort. How can this be done?

Details:

I want to use packetbeat to just capture packets as they are (I dont care much about it's ability to decode layer 7 etc for my use case) and send the packets with compression to a remote http end point? This could be haproxy for example, and from there I can consume packets, dump them into a time series database etc

I am aware that packetbeat can write to local pcap files, but just want to capture and send the packets somewhere and analyze them offline using inhouse tools which understand pcap format. I can ofcourse keep copying pcap files from server for offline analysis but this is too heavy on disk IO.

I am also aware that I can get a JSON doc indicating packet and stash it into ES, but I really dont want to store JSON when all I need is actual packets.

Am I thinking in right direction here? Any thoughts? Is this possible?

2

writing of pcap files in packetbeat is more for debugging purposes. Packetbeat does deep packet inspection and publishes it's results to logstash/elasticsearch.

3

Yes, I understand that. Is there a way to just use packetbeat to ship packets ? Is this something which is on the roadmap? The utility of just having all packets and doing offline analysis on them is huge. I see packetbeat tries to do more, my question is is there any way to configure it do less and just ship packets? A lot like what rpcapd from WinPcap suite does?

4

no, this is out of scope of packetbeat. Tools like rpcapd are better suites for this use-case.

For offline analysis packetbeat can also be driven from files.

Personally I was thinking about having a packetbeat companion tool for pushing the packets to a remote packetbeat instance or via some queuing system, but then I wouldn't write this kind of tool in go.

5

Interesting thoughts. Wouldnt it make sense to do this in a memory safe language like Rust/Go? C would be more efficient but unsafe?

Would love to know what you are thinking about this tool. Happy to pitch in to help with testing etc.

6

Well, I didn't start anything yet, nor will I start anything anytime soonish.

For programming language I'd consider C/C++ (11+) or Rust. When dealing with sniffers on a lower level, ownership management of packets processed becomes more ingrained in the tool itself (we can even do this in go without the need to allocate). Especially when dealing with af_packet, DPDK or even trying to integrate with Napatech and such... it might be somewhat easier if it's not go.

gopacket kind of tries to hide this fact by copying the original packets into another temporarily allocated buffer or providing access to shared memory to one (most recent) packet only.

On the other hand I'm considering to get rid of gopacket based sniffing in packetbeat. Maybe go becomes a viable options then...

7

On the other hand I'm considering to get rid of gopacket based sniffing in packetbeat. Maybe go becomes a viable options then...

Yes. that makes sense. Agree.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.