Filebeats filters under load

Elastic Stack Beats
1

Hi there! It's me again!
Running into some inexplicable behaviour. I am deployed a filebeater, and it spammed millions of messages up to the Kibana server. I identified some garbage lines, and added "exclude_lines:" filters to remove the bastages.

exclude_lines: ["active count - 0","Got 0 messages from sqs queue"]

I then waddle over to Kibana, and sure enough, my message count in now in an acceptable "100's" of records, but to my surprise it seems some of the messages are leaking through the filter:
Message:
9.07.2017 18:07:30 INFO (Starter.java:57) - Thread pool status, core pool size - 6, pool size - 6, active count - 0

Not sure why those are still showing up at all. Yes, they are reduced, but there should be none. If FIlebeats gets to busy could stuff slip through?

2

Problem solved. The same message was sneaking in from another log file.

3

Note:exclude_lines uses regular expression. I prefer to always use single quotes with regexes (well, in general single quotes for strings in YAML based config files).

The exclude_lines is applied right after reading a line (or multiline), but before building the actual event. Alternatively you can use processors globally or per prospector, to do some additional filtering (disadvantage: processors are applied after creating the actual event in memory).

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.