Logstash wildcards in input file path not processing files

Hi, i have a logstash configuration like this:

file {
         path => "/opt/audit_logs_modsec/xx/*/*/*"
    #     path => "/opt/audit_logs_modsec/xx/20171231/20171231-2358/*"
         sincedb_path => "/dev/null"
        start_position => "beginning"
        type => mod_security

        codec => multiline {
          charset => "US-ASCII"
          pattern => "^--[a-fA-F0-9]{8}-Z--$"
          negate => true
          what => previous

logstash starts but never reads any file
if i uncomment the line:

# path => "/opt/audit_logs_modsec/xx/20171231/20171231-2358/*"

and comment the:

path => "/opt/audit_logs_modsec/xx/*/*/*"

it loads data.

What am i missing?


Please edit your post. Select the text of the configuration and click on </> in the toolbar above the edit pane. In the preview pane you should see the text change

    to be
like this

Instead of # and _ and maybe * getting consumed as markup.

Thanks, let me see if its ok now

Can you try

path => "/opt/audit_logs_modsec/xx/**/*"

which should do a recursive search under xx. Some folks have reported an issue with deep globs.

Thanks Badger, i tryed it on your suggestion but i does not work.

if instead of
path => "/opt/audit_logs_modsec/xx/20171231/20171231-2358/*"

if i put:
path => "/opt/audit_logs_modsec/xx/20171231/*/*"

it works and loads files. It seems to be having trouble when there are too many files.


Well, in tail mode, the default limit is that it will only tail 4095 files. Does read mode fit your use case?

Thanks Badger, it has to be with that limit you mention

[root@elksrv conf.d]# find /opt/audit_logs_modsec/xx/20171231/*/* | wc -l
[root@elksrv conf.d]# find /opt/audit_logs_modsec/xx/*/*/* | wc -l
-bash: /usr/bin/find: Argument list too long

This is just a lab, in real configuration i'll be reading with filebeats so i wont have that problem.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.