Unable to input files in multiple directory levels

Elastic Stack Logstash
1

Hello. I'm having trouble configuring the logstash file input plugin with a path pattern to perform a recursive search across multiple levels in a particular directory. Every time I run logstash it never picks up any of the existing files. When I set the file path to a specific file, it reads it. When I set the file path to recursively search the directory level above the files, it reads them. When I set the next level up to be recursively searched, nothing happens. I am using read mode and have cleared the sincedb file. What am I missing?

Here is an example of my directory structure:

/data/extract/sample/012018/a1/file.xml
/data/extract/sample/012018/a2/file.xml
/data/extract/sample/022018/a1/file.xml
/data/extract/sample/032018/a1/file.xml

Here is an example of my input configuration

input {
  file {
    path => "/data/extract/sample/**/*"
    type => "xml"
    file_sort_by => "path"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/var/log/logstash/completed.log"
    sincedb_path => "/var/log/logstash/sincedb.log"
    codec => multiline {
        pattern => "^<\?xml.*\?>"
        negate => true
        what => "previous"
        auto_flush_interval => 1
        max_lines => 300000
    }
  }
}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.