Thanks a lot. I hope we don't have to go with this dirty trick in the future 
Seems i have found the problem, some logs are working ok and some are not. Turns out that the logs that are not working use a circular logging format called clog.
Now the next question is can filebeat handel this changing the config or am i out of luck? Reverting back to syslog will be like having to turn in a ferrari and stapping in a lada 
I had a quick look at how circular logs work, and yes this is at the moment definitively not supported by filebeat: https://doc.pfsense.org/index.php/Why_can't_I_view_view_log_files_with_cat/grep/etc%3F_(clog)
The surprising part for me is:
random binary data may be seen inside the log and entries may appear out of order.
I'm not sure how other log crawlers handle this issue. It seems that the with the clog command, data can be extracted.
For my better understanding: What is the main disadvantage for you using normal log files vs circular logging except the fixed size?
As I understand this correctly, clog format is basically treating the log file as circular buffer (ring buffer). While I'm a fan of having a ring-buffer based logging systems (easy to size, always operates in its limits, no extra IO due to compression when rotation logs), I'm afraid there is no common widely accepted file format for dealing with circular logs, which need to keep some meta-data (hence "random binary data" when using cat) in order to operate correctly. And even on FreeBSD (I haven't used it for years, though) syslog + newsyslog/logrotate seems to be the common setup.
Is it possible to configure PFsense to log to clog + syslog at the same time? In case you have disk space constraints preventing you from using syslog next to clog, you might consider forwarding logs to another machine running syslog (or logstash).
In case you really want to operate from clog files I see another options, kinda of a hack.
In case you have just one clog file to monitor, you can configure filebeat to capture stding and start filebeat like clog -f ... | filebeat -c /etc/filebeat/filebeat.yml. In case you have multiple clog files to monitor you will need a script forwarding clog -f ... output from multiple sources or run one filebeat with stdin enabled only per log file you want to forward.
When reading from log-files filebeat will remember the last read position. When using clog -f and stdin input in filebeat, this feature will be lost. The complete log will be send again. In order to prevent your logs to be duplicate in elasticsearch, consider logstash filters to parse the timestamps and create an unique document id (e.g. from timestamp + message hash).
The main reason to use Filebeat and not syslog is TLS support and better transport (TCP and resume).
Default PfSense uses UDP syslog and for bad internet connections the resume functions of Filebeat is also a reason for going that route.
For now my snort logs are working because they do not use clog. I will try if the clog -f would work for the other log. Maybe someone on the PfSense form knows if clog can be disabled
Just a few days ago @cleesmith published unifiedbeat to read logs from binary files. I don't think it is exactly the same case but it could be interesting to have a look at his implementation: https://github.com/cleesmith/unifiedbeat
Not sure if this is really on topic here, but I'm trying to get the same thing working on pfSense and came across this:
Do a search for "disablesyslogclog"
In theory this will get what you want, but need to be careful of disk space.
I'm going to test tonight. Ideally filebeat would be able to work with the clog setup so you don't ever have to worry bout the space constraint.
I'm going to try and test this out tonight.
Looks like there is a function to disable clog, only i cloud not find it in the gui. Also tried using system tunables but no go.
I do have the snort log working perfectly. there is no filebeat service but using a little shell script i have it running.
Maybe contact the pfsense team to see how to disable clog. p.s if you like i have the working logstash config for snort
Yea, that config prop is not exposed, you'll have to set it manually vi Diag > Edit file > /conf/config.xml (be careful). this will probably take a reboot to function property i would guess, but you might be able to just reload the syslog configuration (worth a try)
Made a pfsense feature request describing the issue: https://redmine.pfsense.org/issues/6155
Has anyone yet found a good way to use filebeat with pfsense's logs in clog format? I am trying to get this to work properly right now.
1 Like
You should remove the preceding '%' on the file /var/etc/syslog.conf
!relayd
*.* **%**/var/log/relayd.log
!hostapd
*.* **%**/var/log/wireless.log
!filterlog
*.* **%**/var/log/filter.log
that will force pfsense to write to standard text files instead of circular logging.
Beware that this change may fill up the disk space quickly, especially if you are logging permited/denied connections.
Regards,
Leo.