Include_lines and exclude_lines not working as expected

Elastic Stack Beats
1

Hi,

I want to exclude a few lines from logs before sending it to logstash (6.6) from Filebeat.
Here is my filebeat.conf:

filebeat.inputs:
- type: log
  paths: 
        - /Users/amit_joshi5/ElasticData/spool/FileReader*.log
  fields: 
        log_type: detectionLogs
        
- type: log 
  paths:
        - /Users/amit_joshi5/ElasticData/spool/FileCsv*.csv
  fields:
        log_type: perfMonLogs

include_lines: ['^FINEST']
exclude_lines: ['^DEBUG']

output.logstash:
  hosts: ["localhost:5044"]

Snippet from Logs:

FINEST: com.jacob.systemio.node FileUpload File: l:\test10\ntlm1.pcap DownloadByteContentSize: 1232 Time taken: 3
DEBUG: com.jacob.systemio.node FileUpload File: l:\test10\ntlm2.pcap DownloadByteContentSize: 213123 Time taken: 2
DEBUG: com.jacob.systemio.node FileUpload File: l:\test10\ntlm4.pcap DownloadByteContentSize: 12312 Time taken: 3
DEBUG: com.jacob.systemio.node FileUpload File: l:\test10\ntlm3.pcap DownloadByteContentSize: 5676 Time taken: 3
DEBUG: com.jacob.systemio.node FileUpload File: l:\test10\ntlm5.pcap DownloadByteContentSize: 4564 Time taken: 3
DEBUG: com.jacob.systemio.node FileUpload File: l:\test10\ntlm.pcap DownloadByteContentSize: 456 Time taken: 5
DEBUG: com.jacob.systemio.node FileUpload File: l:\test10\ntl11.pcap DownloadByteContentSize: 456 Time taken: 6

Based on configuration, it is expected to post only first line to logstash, however, it is posting all the log lines.
I tried multiple things and combinations with include and exclude filter, but none of them worked.
The only intention is to send filtered data to logstash.

2

Any clues ?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.