Logstash wildcards in input file path not processing files

Elastic Stack Logstash
1

Hi, i have a logstash configuration like this:

file {
         path => "/opt/audit_logs_modsec/xx/*/*/*"
    #     path => "/opt/audit_logs_modsec/xx/20171231/20171231-2358/*"
         sincedb_path => "/dev/null"
        start_position => "beginning"
        type => mod_security

        codec => multiline {
          charset => "US-ASCII"
          pattern => "^--[a-fA-F0-9]{8}-Z--$"
          negate => true
          what => previous
        }
      }
    }

logstash starts but never reads any file
if i uncomment the line:

# path => "/opt/audit_logs_modsec/xx/20171231/20171231-2358/*"

and comment the:

path => "/opt/audit_logs_modsec/xx/*/*/*"

it loads data.

What am i missing?

regards

2

Please edit your post. Select the text of the configuration and click on </> in the toolbar above the edit pane. In the preview pane you should see the text change

    to be
like this

Instead of # and _ and maybe * getting consumed as markup.

3

Thanks, let me see if its ok now
regards

4

Can you try

path => "/opt/audit_logs_modsec/xx/**/*"

which should do a recursive search under xx. Some folks have reported an issue with deep globs.

5

Thanks Badger, i tryed it on your suggestion but i does not work.

if instead of
path => "/opt/audit_logs_modsec/xx/20171231/20171231-2358/*"

if i put:
path => "/opt/audit_logs_modsec/xx/20171231/*/*"

it works and loads files. It seems to be having trouble when there are too many files.

regards

6

Well, in tail mode, the default limit is that it will only tail 4095 files. Does read mode fit your use case?

7

Thanks Badger, it has to be with that limit you mention

[root@elksrv conf.d]# find /opt/audit_logs_modsec/xx/20171231/*/* | wc -l
3756
[root@elksrv conf.d]# find /opt/audit_logs_modsec/xx/*/*/* | wc -l
-bash: /usr/bin/find: Argument list too long
0

This is just a lab, in real configuration i'll be reading with filebeats so i wont have that problem.

regards

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.