Hi folks, we are using filebeat 6.8 on Ubuntu 18.04.5 LTS with ESM.
Filebeat is reading from
mysql-audit.log thats configured to log CONNECT and QUERY events. Filebeat sends to a load balancer fronting logstash receivers.
In our desire to reduce the amount of data filebeat sends to logstash without altering what mariadb logs to disk, we are testing the use of
include_lines in the definition for that particular log file in filebeats configuration. It appears to be working as we are now seeing only the events with the matching regex set in filebeat config.
There were concerns expressed that adding the filter may increase CPU usage due to the sheer volume of data being processed from
mysql-audit.log. This doesn't appear to be the case, however we did see a tripling of disk write I/O during the period of testing:
Does filebeat cache to disk log data its read before applying filters?