Filebeat 1.1.0 exclude lines with only CRLF or LF

Elastic Stack Beats
1

Hello,

I have been trying to exclude lines which ONLY contains CRLF or LF i.e. empty lines.
However, I have not found a solution yet

I have tried with exclude_lines: ["^\r?\n$"]
And all the variations I can think of, but I am out of luck!

Sample for testing:
Caused by: org.error.something.something
at at.somewhere
at no.nowhere
at com.google
at org.apache
... 156 more

Remember the last line which should contain either CRLF or LF (depending on the OS)

The filebeat 1.1.0 service is running on Windows Server 2012 R2.

Thank you in advance and thank you for an awesome product stack :slightly_smiling:

2

Isn't that identical to excluding empty lines? Means ["^$"] could work (didn't test)?

3

I have just tested your suggestion and I'm afraid it doesn't seem to work.

4

I still haven't found a solution to this problem.
Anyone have any ideas?

5

Didn't have time yet to look into this. @steffens Do you have an idea?

6

  1. in yaml use single quotes, not double quotes for your patterns

  2. try pattern '^[[:space:]]*$' in case line has unspecified number of whitespace characters. This pattern should match all empty lines and all lines with any number of whitespace characters as in [\t\n\v\f\r ]

1 Like
7

I tried exclude_lines: '^[[:space:]]*$' in 5.3 filebeat config file, still it is not ignoring the empty lines.

8

There is an empty line within an event, due to which multiline pattern is not considering the lines after the empty line. Subsequent lines after the empty line are considered as separate/different events. Any thoughts to consider this empty line also part of the multiline event.

9

Instead of prose, do you have an actual example plus configuration you're using?

The exclude_lines filter is applied after multiline. Is multiline spanning the newlines?

10

Here is the sample log file snippet. You can see a BLANKLINE after line.separator = . So file.encoding = UTF-8 is considered as new event, instead of appending to the event it belongs to.

2017-04-07 01:44:27,684 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: JBoss EAP 7.0.0.GA (WildFly Core 2.1.2.Final-redhat-1) starting
2017-04-07 01:44:27,685 DEBUG [org.jboss.as.config] (MSC service thread 1-1) Configured system properties:
                [Standalone] = 
                awt.toolkit = sun.awt.X11.XToolkit
                line.separator = 
                [BLANK/EMPTY LINE]
                file.encoding = UTF-8

Here is the filebeat configuration.

 exclude_lines: '^[[:space:]]*$'
 multiline.pattern: '^[[:space:]]+|^Caused by:'
multiline.negate: false
multiline.match: after

Let me know if any further information is required. Thanks for looking into this post.

11

I am using 5.3 version of filebeat.

12

I did try to format you post, but not sure formatting is correct. Please properly format logs using the </> button.

why you want file.encoding to be another event? This looks like some kind of structured log you can parse in Logstash/Elasticsearch Ingest Node. Other logs might include more fields...

Instead of matching on space, you might consider matching on timestamp/blank line and set multiline.negate: true.

13

file.encoding need not be another event. But here it is being considered by file beat as another event just because of the presence of BLANK/EMPTY line just before that.

I tried matching the timestamp line and set multiline.negate: true . Now I see all events are being appended and finally seeing only one document.

14

Have used multiline.pattern: '^%{TIMESTAMP_ISO8601}' and tried with multiline.pattern: '^%{ISO8601}' also.

15

given your recent post I guess it's not working yet?

with beats you have to use plain old regexes. e.g. '^\d{4}-\d{2}-\d{2} to match a log line starting with a date.

16

Thank you so much. It did work finally. :slight_smile: