Get maximum performance from on file parser

[mancubus77 (cerberus)]

I have a task to process batch of pcap files generated by monitoring and watchdog scripts. Before I discovered PacktBeat a parsing performed via tshark pdml and a few custom parsers. The stack was ugly, but it was fast enough.
Now I'm trying to replace it with packetbeat, but faced with issue of poor performance while reading pcap from FS. With default pcap processing speed is 3.23m per 100k packets. System has 24 cores and not utilised at all, meanwhile old stack squeeze every cpu cycle.

run
packetbeat -t -e -I dns01.pcap

2018-10-10T20:51:55.146+1100 INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1030,"time":{"ms":1033}},"total":{"ticks":20790,"time":{"ms":20801},"value":0},"user":{"ticks":19760,"time":{"ms":19768}}},"info":{"ephemeral_id":"1d77312b-a2ad-445c-a281-31a938361beb","uptime":{"ms":30018}},"memstats":{"gc_next":88851136,"memory_alloc":74132688,"memory_total":997882096,"rss":144236544}},"dns":{"unmatched_requests":615,"unmatched_responses":1438},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":38514,"active":50,"batches":772,"total":38564},"read":{"bytes":539098},"type":"elasticsearch","write":{"bytes":39962820}},"pipeline":{"clients":14,"events":{"active":4118,"published":42630,"retry":50,"total":42632},"queue":{"acked":38514}}},"system":{"cpu":{"cores":24},"load":{"1":0.44,"15":0.79,"5":0.75,"norm":{"1":0.0183,"15":0.0329,"5":0.0313}}}}}}

Would be appreciated for any advise how to make it faster. Google-Fu didn't help much.

Cheers

[andrewkroh]

Packetbeat's reader isn't designed to take advantage of multiple cores at this time. So the thread that reads the PCAP file is also parsing the data then handing the generated events off to a separate thread.

You might get a boost by turning off flows in your config file and disable any other protocols that you are not interested in.

I'm curious if you are only processing DNS data with this setup or are there other protocols too?

[mancubus77 (cerberus)]

Thanks for your reply Andrew
I've followed your advice, but that didn't help much.
PCAP files generated by watchdog and traffic filters applied on tcpdump. Unfortunately I can not run packetbeat or netflow on remote instance because it's on different security tier. Therefore pcap parsing was a solution.
Looks like I have to look into code to have a look on reader section and findout why it's not loading cores.

Cheers!