Include_lines problem

Elastic Stack Beats
1

Hi,
I am trying to configure Filebeats to only process lines from my logs which match a couple of regexes. Nothing too complicated.

I set it up like this:

include_lines: [".*returned\sthe\sstatus.*",".*information\sfrom\sthe\sgreen\sServer.*"]

But then I cannot start the Filebeat service.

It does however work with:-
include_lines: [".*returned the status.*",".*information from the green Server.*"]

However, for some strange reason lots and lots of other lines are also being caught and sent into ES.

My questions are:-

  1. Why is filebeats borking at the \s in the regex??
  2. What regex implementation does Filebeats actually use?

Thanks.

UPDATE
Bit more testing. I simplified the include_lines to be a single expression:-
include_lines: ["returned the status"]

But I am getting all manner of lines sent into ES. Not just the lines one might expect.
This is on a Windows machine. Can anyone shed any light on why "inlclude_line" is just doing its own thing?

2

Hi,

I have now tried many combinations of regex using:

include_lines and exclude_lines

There appears to be no way to only collect the lines that I want. Every combination I have tried thus far has resulted in completely unexpected lines also being shipped to elasticsearch.

As far as I can tell include_lines and exclude_lines simply do not work. I have reduced this problem down to a very simple scenario where all I want to send to Elasticsearch are lines matching a very simple pattern. No matter what I do, everything is sent.

Can anyone please chime in here and advise if this is a known issue on Windows? PLEASE?

3

What version of Filebeat are you using?

Can you share the config file and sample log file for this scenario? This will help someone reproduce the problem and determine if it is a bug.

4

Hi,

I have tried with 1.2.3 and the latest v5 alpha currently offered as a download (5.0.0-alpha3-windows).

I can share both the config file and an excerpt from the log file, but it will have to be privately. Would you like me to DM you with more details?

Thanks

5

Thank you @andrewkroh the information has been sent by DM.

6

The indentation for the encoding and include_lines options is wrong. They need to be moved 4 spaces to right because they are options associated with the prospector. See the indentation is the default config file: https://github.com/elastic/beats/blob/1.2/filebeat/etc/filebeat.yml#L41

Also include_lines expects a list of strings and not a single string. So enclose the value in ["value"] in brackets.

The encoding you specified for the log file does not match the sample log file you sent. There is no BOM in that file so just remove the encoding setting from your config file.

7

Thanks Andrew, I will try that first thing tomorrow.

I had tried the brackets around the strings and understand that. The issue is most likely the indentation.

Appreciate it.

8

Yes, that was it. Now getting what I need. Thank you so much.

Now that it is only processing the lines I need, however, the filebeats.exe is taking 70% CPU on the host machine. Will need to look into that in the morning.

Thanks again.

9

This topic was automatically closed after 21 days. New replies are no longer allowed.