Does File input work recursively

(Niraj Kumar) #1

Hi Everyone,

I am building up my logstash config and i am using the following file input plugin to ingest gzip files.

input {
file {
	path => "/home/niraj/sync/bigdata-dev"
	tags => "bigdata-dev1"
	codec => "gzip_lines"
	start_position => "beginning"
	sincedb_path => "gzip"
	type => "cloudtrail"

My question here is that the path i am having has multiple sub-directories and each one of these have *.gz files ( cloudtrail data ), will the file input recursively go through each sub-directories?


(Magnus Bäck) #2

That should work if your filename pattern is set up for that with /home/niraj/sync/bigdata-dev/**/*.gz or /home/niraj/sync/bigdata-dev/*/*.gz (not sure which one you need to use). Right now you've configured it to read a single file, /home/niraj/sync/bigdata-dev.

If you bump up the log level Logstash will log what filename patterns expand to.

(Niraj Kumar) #3

@magnusbaeck Thanks for the response. My directory structure is /home/niraj/sync/bigdata-dev/**/.gz.

Do i need to add anything else to my file input?

(Magnus Bäck) #4

Do i need to add anything else to my file input?


I saw that the asterisks in my previous post caused it to get mangled. I've fixed that.

(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.