Packetbeat to analysi offline packet captures

[anilkumar911 (Anil)]

Hi,

I plan to use/modify packetbeat as an alternative to wireshark, to analyse huge trace files ( > 10GB) . I have 2 requirements below:

1. export data from a pcap capture file
2. capture all fields of different protocol layers like mac address, tcp flags etc.

Before getting started, I want to know

1. If there is an existing tool that can accomplish this?
2. Is this fundamentally against the philosophy of packetbeat/elasticsearch ?

Thanks in advance,
Anil

[tudor (Tudor Golubenco)]

It's not against the philosophy of packetbeat, but there are somethings that you need to be aware of:

• We currently have the -I flag for reading pcap files. It's mostly use for development and testing.
• Unlike Wireshark/tshark, Packetbeat does correlation of the requests with the responses, and inserts a single document per request-response pair in ES. This is generally good as it's a lot more convenient to have them correlated once in ES.
• On the other hand, Packetbeat supports a lot less protocols compared to Wireshark/tshark.
• While you can use Packetbeat to read the packets from a pcap file, its implementation assumes that the packets are read in real time, which can cause some issues with timeout transactions (the time is measured from the current time, not from the initial packet's time).
• Depending on which protocols you need, and whether you need correlation or not, tshark might be a better choice for you. Tshark supports a JSON output that can be easily ingested into Elasticsearch using Filebeat.

What use case do you have in mind? If it's for security analysis, Moloch might also be an interesting option for you.

[anilkumar911 (Anil)]

Thanks Tudor for your reply. Moloch was a good find. But what I want is slightly different.

'tshark -T ek ...' to gives json representation of individual packets that can be bulk imported to ES. Is there any project that gives wireahrk like UI for this data in ES ? (since tshark has an option specific to ES, I'm guessing there should already be a viewer for this data)

I want to create a tool to analyse large trace files in order to debug network issues (related to a proxy/load balancer which we support). tshark already handles the capture part of my requirement(performance is not a criteria on the capture side). I will have to create a GUI to view the captured packet data. Do you know of any tool that already does this?

--Anil