Analyzing PCAP's with Zeek(Bro) in offline mode

Elastic Stack Kibana
1

Hi all,

I am using Zeek to analyze the pcaps in the offline mode. To run the pcap i use command "sudo zeek -r /path/to/file.pcap. It generates the logs (dns.log, http.log, ssl.log etc) in the /opt/zeek/logs/current folder. These logs are not in JSON format. In the inline mode with the help of this code redef LogAscii::use_json = T; zeek converts those logs into json which are then ready to be parsed to elasticsearch, which works fine for me. But how to convert those generated logs during offline execution to JSON so that they can be parsed to elasticsearch. I tried to use a logstash filter for conversion but of no use.
So here is how i run:

  • make sure that Kibana and Elasticsearch services are running in the background.
  • if i have to ingest logs to logstash then i enable "use logstash" in filebeat.yml otherwise it is disabled.
  • then run the filebeat using filebeat -e command
  • then finally run zeek -r /path/to/file.pcap
    After that i get this error "ERROR [reader_json] readjson/json.go:57 Error decoding JSON: invalid character # looking for beginning of value" and the reason for this is that the log files format starts with #
    i.e.
    #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path dns #open 2020-11-06-10-18-29 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id rtt query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected #types time string addr port addr port enum count interval string count string count string count string bool bool bool bool count vector[string] vector[interval] bool 1601918839.155711 C5yYobSodHUVqnxP6 192.168.1.10 50158 192.168.1.1 53 udp 23469 0.019095 edge.microsoft.com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 edge-microsoft-com.a-0016.a-msedge.net,a-0016.a-msedge.net,204.79.197.219 2286.000000,46.000000,46.000000 F.
    I would be happy to have some guidance to resolve this issue. Thank You.
2

Filebeat have a module to parse zeek brok json log
You can use json format is zeek bro

3

Thankyou for your reply @ylasri. Zeek module in filebeat works perfectly fine. and all zeek logs are parsing in elasticsearch. but all these are happening in inline mode i.e. I have to use "zeekctl deploy" command and it starts zeek. In that way it captures all the packets passing through the set interface i.e. eth0.
I want to use offline mode for pcap. I do get the logs in the current directory but as i mentioned that those are not in json format, so the filebeat throws the error when i wanted to get them in elasticsearch using filebeat. so i don't get the desired results in elasticsearch.
Is there any way to convert those generated logs into json format using logstash filter? or to have them in json during the execution of pcap?

4

Basically this is your log format wish looks like a CSV based that can be easily parsed by logstash using csv filter, or any other filter like grok or dissect

#separator \x09 
#set_separator , 
#empty_field (empty) 
#unset_field - 
#path dns 
#open 2020-11-06-10-18-29 
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id rtt query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected 
#types time string addr port addr port enum count interval string count string count string count string bool bool bool bool count vector[string] vector[interval] bool 
1601918839.155711 C5yYobSodHUVqnxP6 192.168.1.10 50158 192.168.1.1 53 udp 23469 0.019095 edge.microsoft.com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 edge-microsoft-com.a-0016.a-msedge.net,a-0016.a-msedge.net,204.79.197.219 2286.000000,46.000000,46.000000 F
1601918839.155711 C5yYobSodHUVqnxP6 192.168.1.10 50158 192.168.1.1 53 udp 23469 0.019095 edge.microsoft.com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 edge-microsoft-com.a-0016.a-msedge.net,a-0016.a-msedge.net,204.79.197.219 2286.000000,46.000000,46.000000 F
1601918839.155711 C5yYobSodHUVqnxP6 192.168.1.10 50158 192.168.1.1 53 udp 23469 0.019095 edge.microsoft.com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 edge-microsoft-com.a-0016.a-msedge.net,a-0016.a-msedge.net,204.79.197.219 2286.000000,46.000000,46.000000 F
1601918839.155711 C5yYobSodHUVqnxP6 192.168.1.10 50158 192.168.1.1 53 udp 23469 0.019095 edge.microsoft.com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 edge-microsoft-com.a-0016.a-msedge.net,a-0016.a-msedge.net,204.79.197.219 2286.000000,46.000000,46.000000 F
1601918839.155711 C5yYobSodHUVqnxP6 192.168.1.10 50158 192.168.1.1 53 udp 23469 0.019095 edge.microsoft.com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 edge-microsoft-com.a-0016.a-msedge.net,a-0016.a-msedge.net,204.79.197.219 2286.000000,46.000000,46.000000 F

You need first to drop all events (lines) that start by # and then apply your filter

if [message] =~ /^#/ {
    drop { }
  }

Then apply your filter depending of the type of the input file

if [type] == "bro-dns-logs" {
    csv {
      columns => ["ts", "uid", "id.orig_h", "id.orig_p", "id.resp_h", "id.resp_p", "proto", "trans_id", "rtt", "query", "qclass", "qclass_name", "qtype", "qtype_name", "rcode", "rcode_name", "AA", "TC", "RD", "RA", "Z", "answers", "TTLs", "rejected"]
      separator => "	"
    }

Hope this can help

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.