Can I use conditional logic to filter log lines with filebeat?

I previously asked how I could send only a sample of my logs with filebeat: Ship only a percentage of logs to logstash

Im filtering iis request logs and the volume is very high, i want a representative sample of all of those requests.

Right now I am doing this with nxlog and I am curious if this is possible with filebeat:
Exec if (get_var('count') == undef) set_var('count',0);
Exec set_var('count', get_var('count') + 1);
Exec if (get_var('count')%20 != 0) drop();

Im setting a counter for each log line and dropping it if it is not divisible by 20 (so effectively 5%).

How can I do this with filebeat?

You'd have to send them to LS and then run a ruby filter with that logic.

Well logstash can actually just drop a percentage: https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html#plugins-filters-drop-percentage

But thats not my problem. I need to filter these logs on the endpoints side because of bandwidth concerns.

Sorry I already asked a similar question, but I thought if I included how I'm doing this with nxlog, someone might know an analogue for filebeat.

There is currently no way to do this in Filebeat.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.