Exclude_lines in filebeat does not work?

Elastic Stack Beats
1

Hi,

I have log records like:

2022-12-12T07:07:07.007


***********************************

bla bla bla

To exclude these line I use exclude_lines: ['^\n$','^\*+$'] in my filebeat configuration.
But anyway I get empty lines and asterisk lines to ES.

What's wrong?

2

I used to have data in below format.

=============================================
# Timestamp : 2019-11-11T09:02:08.1591376Z
# SourceId  : 64f8a4d1-e1ec-5a27-52e2-5f6a013
# Level     : Informational

# serverName : PEPWAP12241
# systemId : 5962
# processorName : NACanadaIRSS_CI
# methodName : ProcessFiles
# userGpid : NACANADAIRSSProcessor
# information : Document Type #3 was set to be processed.

# Message   : Information [System# 5,962 Processor 'NACanadaIRSS_CI' Method 'ProcessFiles' ] :: UserGpid# NACANADAIRSSProcessor : Information - Document Type #3 was set to be processed.
# Payload   : [serverName : PEPWAP12241][systemId : 5962][processorName : NACanadaIRSS_CI][methodName : ProcessFiles][userGpid : NACANADAIRSSProcessor][information : Document Type #3 was set to be processed.]
==============================================

And my filebeat config looks in this way and it worked for me.

#=========================== Filebeat inputs =============================
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
  enabled: true
  exclude_lines: ['^==============================================']
  fields: 
    log_topics: nextgen-canada
  multiline.match: after
  multiline.negate: true
  multiline.pattern: "^#\\sTimestamp"
  paths: 
    - "D:\\Logging Service\\IRSSNextGen-NA\\*.log"
3

I suppose, the problem is in multiline settings.

Because multiline combine message into a single line before applying exclude_lines filter

4

Yes. Try out in the way I've suggested and see if it works for you.

5

It will exclude all lines containing that expression?

6

exclude_lines: ['^==============================================']

The "^" symbol in the beginning says, starting with.

So any line starting with the expression which I've given will be excluded from my file.

7

Because you have another structure of the log.

After applying multiline I have something like this

2022-12-12T07:07:07.007\n\n\n***********************************\nbla bla bla
8

Please share your log data of 2 - 3 lines and also send you filebeat config, so that I can look at it and can let you know

9

All sufficient and necessary information is in my first post.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.