Filebeat- syntax for regex file exclusion

Elastic Stack Beats
1

I am having issues figuring out the syntax for excluding a list of files in a directory. They all come or none come.

Here is the file listing in the directory I am pushing.

total 232676
drwxr-xr-x 6 deploy deploy     4096 Oct 27 07:14 ..
-rw-rw-r-- 1 deploy deploy 10485962 Nov  5 02:40 EnduranceManagementService31-tracer.8.log
-rw-rw-r-- 1 deploy deploy 10485944 Nov  5 06:10 EnduranceManagementService31-tracer.7.log
-rw-rw-r-- 1 deploy deploy 10486260 Nov  5 09:31 EnduranceManagementService31-tracer.6.log
-rw-rw-r-- 1 deploy deploy 10486185 Nov  5 12:11 EnduranceManagementService31-tracer.5.log
-rw-rw-r-- 1 deploy deploy 10486019 Nov  5 13:33 EnduranceManagementService31-tracer.4.log
-rw-rw-r-- 1 deploy deploy 10559538 Nov  5 14:32 EnduranceManagementService31.10.log
-rw-rw-r-- 1 deploy deploy 10560766 Nov  5 14:32 EnduranceManagementService31.9.log
-rw-rw-r-- 1 deploy deploy 10486314 Nov  5 14:51 EnduranceManagementService31-tracer.3.log
-rw-rw-r-- 1 deploy deploy 10486576 Nov  5 15:03 EnduranceManagementService31.8.log
-rw-rw-r-- 1 deploy deploy 10487941 Nov  5 16:30 EnduranceManagementService31-tracer.2.log
-rw-rw-r-- 1 deploy deploy 10485860 Nov  5 16:31 EnduranceManagementService31.7.log
-rw-rw-r-- 1 deploy deploy 10490736 Nov  5 17:35 EnduranceManagementService31.6.log
-rw-rw-r-- 1 deploy deploy 10559632 Nov  5 17:36 EnduranceManagementService31.5.log
-rw-rw-r-- 1 deploy deploy 10559724 Nov  5 17:36 EnduranceManagementService31.4.log
-rw-rw-r-- 1 deploy deploy 10559267 Nov  5 17:36 EnduranceManagementService31.3.log
-rw-rw-r-- 1 deploy deploy 10527715 Nov  5 17:49 EnduranceManagementService31.2.log
-rw-rw-r-- 1 deploy deploy 10486167 Nov  5 18:22 EnduranceManagementService31-tracer.1.log
-rw-rw-r-- 1 deploy deploy 10548899 Nov  5 19:10 EnduranceManagementService31.1.log
drwxr-xr-x 2 deploy deploy     4096 Nov  5 19:10 .
-rw-rw-r-- 1 deploy deploy  1651540 Nov  5 19:34 EnduranceManagementService31-gc.log
-rw-rw-r-- 1 deploy deploy  7954738 Nov  5 19:35 EnduranceManagementService31-tracer.log
-rw-rw-r-- 1 deploy deploy 34027868 Nov  5 19:35 EnduranceManagementService31-access.log
-rw-rw-r-- 1 deploy deploy  5266139 Nov  5 19:35 EnduranceManagementService31.log

my filebeat config file looks like this.

  paths:
    - /opt/active/logs/*.log
  exclude_files: ['-gc\.log$','-tracer.*']

which works fine because I am not getting any of the tracer or gc files.

the issue is I am trying to exclude ANYTHING that has the pattern of *.?.log or *.??.log like you would look up using the ls command - I have tried a few things like.

'*\..\.log'
'.\*\..\.log'
`*.?.log'

i read somewhere that . is seen as any character so I need to escape it if I want the parse to see the .

if I put any of the listings above it stops all logging.
please help this is a bit convoluted for my brain.

I use this configuration on many servers and the filename prefixes change so that has to be like

Anything.[1-9].log or Anything.10.log should be excluded

I tried this looking at some regex documentation- but it blocks everything.

^[[:alnum:]]+.[[:digit:]]+.log$

exclusion line looks like this- nothing is getting through .

exclude_files: ['-gc\.log$','-tracer.*','^[[:alnum:]]+\.[[:digit:]]+\.log$']

is that because the syntax within each of the items below is different? please help

2

Hey @txmrlevine,

I think you can use this pattern to exclude the rotated files: \.\d+\.log$.

But if all the files you are trying to ignore are rotated files, then maybe other option you can try is to use ignore_older.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.