Filebeat on FreeBSD / PFsense

and statring again (and it picking up the cange):
6/01/05 12:43:50.024396 prospector.go:259: DBG Check file for harvesting: /var/log/system.log
2016/01/05 12:43:50.024546 prospector.go:382: DBG Update existing file for harvesting: /var/log/system.log
2016/01/05 12:43:50.024665 prospector.go:425: DBG Not harvesting, file didn't change: /var/log/system.log
2016/01/05 12:43:50.024780 prospector.go:246: DBG scan path /var/log/snort/snort_*/alert
2016/01/05 12:43:50.022743 util.go:20: DBG full line read
2016/01/05 12:43:50.024958 util.go:20: DBG full line read
2016/01/05 12:43:50.024972 util.go:20: DBG full line read
2016/01/05 12:43:50.024986 util.go:20: DBG full line read
2016/01/05 12:43:50.025003 util.go:20: DBG full line read
2016/01/05 12:43:50.025051 util.go:20: DBG full line read
2016/01/05 12:43:50.025067 util.go:20: DBG full line read
2016/01/05 12:43:50.036113 reader.go:138: DBG End of file reached: /var/log/system.log; Backoff now.
2016/01/05 12:43:50.039518 reader.go:138: DBG End of file reached: /var/log/filter.log; Backoff now.
2016/01/05 12:43:50.041737 reader.go:138: DBG End of file reached: /var/log/dhcpd.log; Backoff now.
2016/01/05 12:43:51.049504 reader.go:138: DBG End of file reached: /var/log/system.log; Backoff now.
2016/01/05 12:43:51.049559 reader.go:138: DBG End of file reached: /var/log/filter.log; Backoff now.
2016/01/05 12:43:51.049577 reader.go:138: DBG End of file reached: /var/log/dhcpd.log; Backoff now.
2016/01/05 12:43:52.538229 spooler.go:97: DBG Flushing spooler because of timeout. Events flushed: 26
2016/01/05 12:43:52.538387 preprocess.go:37: DBG Start Preprocessing
2016/01/05 12:43:52.538562 publish.go:102: DBG Publish: {
"@timestamp": "2016-01-05T12:43:50.021Z",
"beat": {
"hostname": "fw01klantnaam.domain.local",
"name": "fw01klantnaam.domain.local"
},
"count": 1,
"fields": null,
"input_type": "log",
"message": "Jan 5 12:59:54 fw01klantnaam filterlog: 74,16777216,,1000002665,em1,match,pass,out,4,0x0,,64,44471,0,DF,6,tcp,60,172.21.10.199,172.21.10.53,8549,5044,0,S,163065722,,65228,,mss;nop;wscale;sackOK;TS",
"offset": 16793,
"source": "/var/log/filter.log",
"type": "log"

That is quite strange. Based on your config in the beginning I assume you don't use any special scan_frequency or ignore_older? There is also no file rotation happening?

Can you check when filebeat is not sending any updates anymore what the last modified date of the files was? And what happens if you update the file manually?

Thanks for your patience and help.

There is no scan_frequency or ignore_older, but i did try those in differect setting but non helpt. The files can rotate but i did check that this is not the case during testing.

The last moified date will change during a running filebeat, manualy addding a entry will not trigger.

What is the file system type your are using?

Running on FreeBSD 10.1 using UFS.

In the meanwhile i did some more testing because i'am running on vmware. But a physical system installed on a shuttle pc is giving the same result.

I think I really have to try it out myself on FreeBSD. I get more and more the feeling, that it is a FreeBSD and not a general issue. One guess I have is that there is something wrong either with inode changes or reading modification changes, but that doesn't explain why it works on restart.

One very interesting thing would be to run our unit and system tests on freebsd. For the unit-tests, could you run make unit-tests inside the filebeat directory? For the full testsuite, run make testsuite, but it requires python. It would be interesting if these tests pass.

I also did a dirty trick, but this works. A Script kills filebeat and starts it again evenry minute. This works and i am getting the logs.

Thanks a lot. I hope we don't have to go with this dirty trick in the future :wink:

Seems i have found the problem, some logs are working ok and some are not. Turns out that the logs that are not working use a circular logging format called clog.

Now the next question is can filebeat handel this changing the config or am i out of luck? Reverting back to syslog will be like having to turn in a ferrari and stapping in a lada :frowning:

I had a quick look at how circular logs work, and yes this is at the moment definitively not supported by filebeat: https://doc.pfsense.org/index.php/Why_can't_I_view_view_log_files_with_cat/grep/etc%3F_(clog)

The surprising part for me is:

random binary data may be seen inside the log and entries may appear out of order.

I'm not sure how other log crawlers handle this issue. It seems that the with the clog command, data can be extracted.

For my better understanding: What is the main disadvantage for you using normal log files vs circular logging except the fixed size?

As I understand this correctly, clog format is basically treating the log file as circular buffer (ring buffer). While I'm a fan of having a ring-buffer based logging systems (easy to size, always operates in its limits, no extra IO due to compression when rotation logs), I'm afraid there is no common widely accepted file format for dealing with circular logs, which need to keep some meta-data (hence "random binary data" when using cat) in order to operate correctly. And even on FreeBSD (I haven't used it for years, though) syslog + newsyslog/logrotate seems to be the common setup.

Is it possible to configure PFsense to log to clog + syslog at the same time? In case you have disk space constraints preventing you from using syslog next to clog, you might consider forwarding logs to another machine running syslog (or logstash).

In case you really want to operate from clog files I see another options, kinda of a hack.
In case you have just one clog file to monitor, you can configure filebeat to capture stding and start filebeat like clog -f ... | filebeat -c /etc/filebeat/filebeat.yml. In case you have multiple clog files to monitor you will need a script forwarding clog -f ... output from multiple sources or run one filebeat with stdin enabled only per log file you want to forward.

When reading from log-files filebeat will remember the last read position. When using clog -f and stdin input in filebeat, this feature will be lost. The complete log will be send again. In order to prevent your logs to be duplicate in elasticsearch, consider logstash filters to parse the timestamps and create an unique document id (e.g. from timestamp + message hash).

The main reason to use Filebeat and not syslog is TLS support and better transport (TCP and resume).

Default PfSense uses UDP syslog and for bad internet connections the resume functions of Filebeat is also a reason for going that route.

For now my snort logs are working because they do not use clog. I will try if the clog -f would work for the other log. Maybe someone on the PfSense form knows if clog can be disabled

Just a few days ago @cleesmith published unifiedbeat to read logs from binary files. I don't think it is exactly the same case but it could be interesting to have a look at his implementation: https://github.com/cleesmith/unifiedbeat

Not sure if this is really on topic here, but I'm trying to get the same thing working on pfSense and came across this:

https://redmine.pfsense.org/projects/pfsense/repository/revisions/master/entry/src/etc/inc/system.inc

Do a search for "disablesyslogclog"

In theory this will get what you want, but need to be careful of disk space.

I'm going to test tonight. Ideally filebeat would be able to work with the clog setup so you don't ever have to worry bout the space constraint.

I'm going to try and test this out tonight.

Looks like there is a function to disable clog, only i cloud not find it in the gui. Also tried using system tunables but no go.

I do have the snort log working perfectly. there is no filebeat service but using a little shell script i have it running.

Maybe contact the pfsense team to see how to disable clog. p.s if you like i have the working logstash config for snort

Yea, that config prop is not exposed, you'll have to set it manually vi Diag > Edit file > /conf/config.xml (be careful). this will probably take a reboot to function property i would guess, but you might be able to just reload the syslog configuration (worth a try)

Made a pfsense feature request describing the issue: https://redmine.pfsense.org/issues/6155

Has anyone yet found a good way to use filebeat with pfsense's logs in clog format? I am trying to get this to work properly right now.

1 Like

You should remove the preceding '%' on the file /var/etc/syslog.conf

!relayd
*.*                                                             **%**/var/log/relayd.log
!hostapd
*.*                                                             **%**/var/log/wireless.log
!filterlog
*.*                                                             **%**/var/log/filter.log

that will force pfsense to write to standard text files instead of circular logging.
Beware that this change may fill up the disk space quickly, especially if you are logging permited/denied connections.

Regards,
Leo.