I previously asked how I could send only a sample of my logs with filebeat: Ship only a percentage of logs to logstash
Im filtering iis request logs and the volume is very high, i want a representative sample of all of those requests.
Right now I am doing this with nxlog and I am curious if this is possible with filebeat:
Exec if (get_var('count') == undef) set_var('count',0);
Exec set_var('count', get_var('count') + 1);
Exec if (get_var('count')%20 != 0) drop();
Im setting a counter for each log line and dropping it if it is not divisible by 20 (so effectively 5%).
How can I do this with filebeat?