Using Filebeat with Zeek (issue with configuration)

[michaelberg]

Good afternoon everyone!

Having a bit of an issue here with configuration and I hoping someone can kick me in the right direction ...

Software Running:

• Latest compiled version of Zeek on a Fedora 30 Server
• Filebeat 7.2.0 running on the same server as Zeek

Here's the issue:

When running Filebeat and the Zeek module to pull in the logs the Filebeat module defaults to wanting to pull the Zeek logs from "/var/log/bro/current/" (see attached from the Filebeat output)

The problem is that when running Zeek itself it defaults to putting the logs in "/usr/local/zeek/logs" (see attached from terminal session)

So ... I assume at this stage that this is where the "var.paths" option in the Filebeat Zeek module would be put into action .... so I modify the variable as shown in the attached:

However, when I try and fire up Filebeat with this variable change in place Filebeat bombs right away and I get the attached error:

If anyone has a tip they could pass along or has perhaps encountered the same thing I'd sure appreciate some feedback on this!

NOTE: (Added 28-JUL-2019) One thing I think it could be is that I have not converted the Zeek logs into JSON format yet. I tried a test of leaving everything as it was with Filebeat, creating a "/var/log/bro/current/" directory and making a copy of all the current text Zeek logs in there and then just turning Filebeat loose. It immediately starting throwing a bunch of JSON related errors immediately. I'll chase down this idea for now and look into how I can get Zeek to output their logs in JSON. In the meantime if someone else has some other feed back to share I would be most grateful .. Thank you!

NOTE2: (Added 28-JUL-2019) Ok ... I sorted out how to finally get ZEEK to create all it's files in JSON format ... however when starting up Filebeat it's still stopping and producing the "Fileset zeek/var is configured but doesn't exist" error ... I can definitely confirm that all the logs are definitely in JSON format and the var.paths variable for the Zeek module in filebeat.yml is set to the path to where Zeek is producing all the current logs.

NOTE3: (Added 28-JUL-2019) Oh boy ... what a difference an indent makes I feel a tad silly however the previous "Fileset zeek/var is configured but doesn't exist" error was caused by me not properly indenting the var.paths variable in the filebeat.yml Zeek module. Oh well ... that issue solved ... so I have the Zeek logs being created in JSON format, I have the var.paths variable properly set now and I'm still getting the attached error when starting up.

I am wondering if there is some possible incompatibilities between Filebeat 7.2.0 and the latest compiled version of Zeek... I pulled the source for Zeek from their GitHub as of July 21st, 2019 ... thanks in advanced for any feedback! Cheers!

2019-07-28T07:00:00Z
I checked out another post here on the board and the Elastic Team member seemed to suggest that the var.paths variable be set on each item in the Zeek module so that was done as shown here.

Also ... for informational sake here's a shot showing the path and contents of the Zeek Logs "current" directory and snippet of the conn.log showing it's indeed in JSON format.

However, even after making these changes I am still getting the attached error ...

Thanks so much everyone! Cheers!

[ebnx (Charles)]

I think var.paths needs to be an array of paths:

connection:
    enabled: true
    var.paths: ["/var/log/bro/current/conn.log"]
http:
    enabled: true
    var.paths: ["/var/log/bro/current/http.log"]

That works for me. Hope this helps!

[myke (Michael Baer)]

Hi Michael,

Did you ever find a solution to your issue?

I'm asking because I appear to have the same problem. I'm trying to load bro log files into elasticsearch via filebeat and its zeek module. The same error messages show up and the bro files are not loading into elasticsearch. (The system software is only slightly different: Centos 7.6, Elastic 7.3, Bro 2.6)

[michaelberg]

Good Morning and my apologies for taking a bit to get back to you ...

I ended up kinda "giving up" in a way ... and I just changed Zeek to start generating it's log to the place where Filebeat was expecting to see them as I couldn't get the var.paths option to work and I had to "move on" ...

Ever since I did that its worked like a charm ... in the end I guess it doesn't really matter where Zeek stores its logs as long as it generates them successfully and Filebeat can find them

[myke (Michael Baer)]

Thanks for the response!, i actually thought it was pretty quick.

FWIW, an hour or two after I posted, elasticsearch starting loading bro data. Unfortunately, I'm not sure why. I did a couple things awhile before it started working including upgrading to Elastic 7.3.1 and playing with the zeek module config file, current example:

- module: zeek
  # All logs
  connection:
    enabled: true
    var.paths: ["/home/zeek/logs/current/conn.log"]
  dns: 
    enabled: true
    var.paths: ["/home/zeek/logs/current/dns.log"]

etc.

I'd be happier if I'd seen a more clear cut reason for the fix or more helpful error message, but at least it's working.

[michaelberg]

I think the big difference with your config file was wrapping the path names in [""] .... I don't think I did that ...

My setup is working and cranking away as we speak so I don't think I'll mess with it at this stage but great to know you got it working!

Cheers!