Filebeat creating write disk i/o when filtering

Elastic Stack Beats
1

Hi folks, we are using filebeat 6.8 on Ubuntu 18.04.5 LTS with ESM.
Filebeat is reading from mysql-audit.log thats configured to log CONNECT and QUERY events. Filebeat sends to a load balancer fronting logstash receivers.

In our desire to reduce the amount of data filebeat sends to logstash without altering what mariadb logs to disk, we are testing the use of include_lines in the definition for that particular log file in filebeats configuration. It appears to be working as we are now seeing only the events with the matching regex set in filebeat config.

There were concerns expressed that adding the filter may increase CPU usage due to the sheer volume of data being processed from mysql-audit.log. This doesn't appear to be the case, however we did see a tripling of disk write I/O during the period of testing:

Screen Shot 2023-09-07 at 12.55.01 PM

Does filebeat cache to disk log data its read before applying filters?

2

We have determined so far that the disk write I/O is occurring on the root disk from which filebeat is configured and executed. When we stop filebeat, disk write I/O returns to level prior to starting it. Confirmed which disk using iostat. strace doesn't show any write activity.

Any ideas?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.