Get file name from s3 file

Elastic Stack Logstash
1

Hi
i have s3 bucket from where i taking csv file and uploading in to ES .
i stuck on how to get file name which i have in s3 , beacuse i am using s3 file name as Index name .
My config file -

input{
	s3{
	bucket => "***"
	access_key_id => "**"
	secret_access_key => "***"
	region => "**"

}
}

filter {
	csv {
		separator => ","
		columns  => [ "name","uid","startdate","enddate"]

	}
	mutate {convert =>[uid , "integer"]}
	
	  date {
      match => [ "startdate", "ISO8601", "YYYY/MM/dd HH:mm:ss","YYYY/MM/dd HH:mm" ]
      target => "startdate"
      locale => "en"
	  timezone => "Asia/Dubai"
    }
	
	  date {
      match => [ "enddate", "ISO8601", "YYYY/MM/dd HH:mm","YYYY/MM/dd HH:mm:s" ]
      target => "enddate"
      locale => "en"
	  timezone => "Asia/Dubai"
    }
	  mutate {
    add_field => {
      "index_name" => "%{[@metadata][s3][key]}"
    }
   
  }

  }

output {
	elasticsearch {
		hosts => "localhost"
		index => "%{index_name}"
	}
	stdout{}

}

Any suggestion

2

Assuming that the S3 path is available in a field in the event you can use a grok filter to extract the name. How to extract a filename from a full path has been asked several times before so you should be able to find examples in the archives.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.