Using Filebeat with Zeek (issue with configuration)

Good afternoon everyone!

Having a bit of an issue here with configuration and I hoping someone can kick me in the right direction ...

Software Running:

  • Latest compiled version of Zeek on a Fedora 30 Server
  • Filebeat 7.2.0 running on the same server as Zeek

Here's the issue:

When running Filebeat and the Zeek module to pull in the logs the Filebeat module defaults to wanting to pull the Zeek logs from "/var/log/bro/current/" (see attached from the Filebeat output)

The problem is that when running Zeek itself it defaults to putting the logs in "/usr/local/zeek/logs" (see attached from terminal session)

So ... I assume at this stage that this is where the "var.paths" option in the Filebeat Zeek module would be put into action .... so I modify the variable as shown in the attached:

However, when I try and fire up Filebeat with this variable change in place Filebeat bombs right away and I get the attached error:

If anyone has a tip they could pass along or has perhaps encountered the same thing I'd sure appreciate some feedback on this!

NOTE: (Added 28-JUL-2019) One thing I think it could be is that I have not converted the Zeek logs into JSON format yet. I tried a test of leaving everything as it was with Filebeat, creating a "/var/log/bro/current/" directory and making a copy of all the current text Zeek logs in there and then just turning Filebeat loose. It immediately starting throwing a bunch of JSON related errors immediately. I'll chase down this idea for now and look into how I can get Zeek to output their logs in JSON. In the meantime if someone else has some other feed back to share I would be most grateful .. Thank you!

NOTE2: (Added 28-JUL-2019) Ok ... I sorted out how to finally get ZEEK to create all it's files in JSON format ... however when starting up Filebeat it's still stopping and producing the "Fileset zeek/var is configured but doesn't exist" error ... I can definitely confirm that all the logs are definitely in JSON format and the var.paths variable for the Zeek module in filebeat.yml is set to the path to where Zeek is producing all the current logs.

NOTE3: (Added 28-JUL-2019) Oh boy ... what a difference an indent makes :slight_smile: I feel a tad silly however the previous "Fileset zeek/var is configured but doesn't exist" error was caused by me not properly indenting the var.paths variable in the filebeat.yml Zeek module. Oh well ... that issue solved ... so I have the Zeek logs being created in JSON format, I have the var.paths variable properly set now and I'm still getting the attached error when starting up.

I am wondering if there is some possible incompatibilities between Filebeat 7.2.0 and the latest compiled version of Zeek... I pulled the source for Zeek from their GitHub as of July 21st, 2019 ... thanks in advanced for any feedback! Cheers!

I checked out another post here on the board and the Elastic Team member seemed to suggest that the var.paths variable be set on each item in the Zeek module so that was done as shown here.


Also ... for informational sake here's a shot showing the path and contents of the Zeek Logs "current" directory and snippet of the conn.log showing it's indeed in JSON format.


However, even after making these changes I am still getting the attached error ...

Thanks so much everyone! Cheers!