Filebeat not excluding lines?

Elastic Stack Beats
1

Running FileBeat 5.1

filebeat.prospectors:
- input_type: log
  paths:
    - c:\\DNSLogging\\trace.log
  include_lines: ['^[0-9]{1,2}\/[0-9]{1,2}\/[0-9]{4}\s{1,2}[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}\s{1,2}', '^\\s{2}Msg\\slength']
  exclude_lines: ['\(\d+\)some\(\d+\)domain\(\d+\)com\(\d+\)\\n']
  multiline.pattern: ^\d{1,2}\/\d{1,2}\/\d{4}\s{1,2}\d{1,2}:\d{1,2}:\d{1,2}\s{1,2}AM|PM
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 7

I want to exclude internal lookups.
But (n)some(n)domain(n)com(n) is still being included in the output.
My understanding is that the multiline should be put to a single line first. Then the include_lines should trigger. Then exclude lines should go last.
Any idea what might be wrong?

3

Could you share some example logs that you are processing? Also enabling debug logging for filebeat could help you to detect the problem.

4

Sure, I'll post something when I get back to work next week, thanks.

5

19/12/2016 8:10:57 AM 0AE0 PACKET 000000F7530B58F0 UDP Rcv 127.0.0.1 856b Q [0001 D NOERROR] A (6)server(4)some(6)domain(3)com(0)
UDP question info at 000000F7530B58F0
Socket = 448
Remote addr 127.0.0.1, port 56279
Time Query=1374460, Queued=0, Expire=0
Buf length = 0x0fa0 (4000)
Msg length = 0x002c (44)
Message:
XID 0x856b
Flags 0x0100
QR 0 (QUESTION)
OPCODE 0 (QUERY)
AA 0
TC 0
RD 1
RA 0
Z 0
CD 0
AD 0
RCODE 0 (NOERROR)
QCOUNT 1
ACOUNT 0
NSCOUNT 0
ARCOUNT 0
QUESTION SECTION:
Offset = 0x000c, RR count = 0
Name "(6)server(4)some(6)domain(3)com(0)"
QTYPE A (1)
QCLASS 1
ANSWER SECTION:
empty
AUTHORITY SECTION:
empty
ADDITIONAL SECTION:
empty

I only care about the lines up to message length.
The resultant single line message looks something like this:

19/12/2016 8:11:52 AM 0AE0 PACKET 000000F753280310 UDP Rcv 127.0.0.1 fcca Q [0001 D NOERROR] A (6)server(4)some(6)domain(3)com(0)\nUDP question info at 000000F753280310\n Socket = 448\n Remote addr 127.0.0.1, port 59602\n Time Query=1374515, Queued=0, Expire=0\n Buf length = 0x0fa0 (4000)\n Msg length = 0x0029 (41)

So if that message "contains (n)some(n)domain(n)com(n)\n" I want to exclude it.

6

If I understand you correctly, if it contains something domain related, you want to exclude the complete event, correct? Perhaps you can try the playground here to make sure your regexp are correct: https://www.elastic.co/guide/en/beats/filebeat/5.1/multiline-examples.html Regexps are tricky :wink:

7

Found the issue with this one.
Regex didn't like the '\n'

I am now using the following regex:
exclude_lines: ['(\d+)(?i:somedomain)(\d+)(?i:com)(\d+)(?i:au)(0)']

8

Glad you found a solution and thanks for sharing it.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.