Can I use conditional logic to filter log lines with filebeat?

Elastic Stack Beats
1

I previously asked how I could send only a sample of my logs with filebeat: Ship only a percentage of logs to logstash

Im filtering iis request logs and the volume is very high, i want a representative sample of all of those requests.

Right now I am doing this with nxlog and I am curious if this is possible with filebeat:
Exec if (get_var('count') == undef) set_var('count',0);
Exec set_var('count', get_var('count') + 1);
Exec if (get_var('count')%20 != 0) drop();

Im setting a counter for each log line and dropping it if it is not divisible by 20 (so effectively 5%).

How can I do this with filebeat?

2

You'd have to send them to LS and then run a ruby filter with that logic.

3

Well logstash can actually just drop a percentage: https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html#plugins-filters-drop-percentage

But thats not my problem. I need to filter these logs on the endpoints side because of bandwidth concerns.

Sorry I already asked a similar question, but I thought if I included how I'm doing this with nxlog, someone might know an analogue for filebeat.

4

There is currently no way to do this in Filebeat.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.