Does File input work recursively

[niraj_kumar]

Hi Everyone,

I am building up my logstash config and i am using the following file input plugin to ingest gzip files.

input {
file {
	path => "/home/niraj/sync/bigdata-dev"
	tags => "bigdata-dev1"
	codec => "gzip_lines"
	start_position => "beginning"
	sincedb_path => "gzip"
	type => "cloudtrail"
  }
}

My question here is that the path i am having has multiple sub-directories and each one of these have *.gz files ( cloudtrail data ), will the file input recursively go through each sub-directories?

--
Niraj

[magnusbaeck (Magnus Bäck)]

That should work if your filename pattern is set up for that with /home/niraj/sync/bigdata-dev/**/*.gz or /home/niraj/sync/bigdata-dev/*/*.gz (not sure which one you need to use). Right now you've configured it to read a single file, /home/niraj/sync/bigdata-dev.

If you bump up the log level Logstash will log what filename patterns expand to.

[niraj_kumar]

@magnusbaeck Thanks for the response. My directory structure is /home/niraj/sync/bigdata-dev/**/.gz.

Do i need to add anything else to my file input?

[magnusbaeck (Magnus Bäck)]

Do i need to add anything else to my file input?

No.

I saw that the asterisks in my previous post caused it to get mangled. I've fixed that.