Hello All,
I have replace my old input with Amazon S3 input because i Need to read CSVs from S3 bucket instead of my local drive.
Old Input syntax (Read CSVs from my local)
input
{
file
{
sincedb_path => '/dev/null'
path => "C:/logstash-2.3.1/logstash-2.3.1/abc/allcsv/*.csv"
start_position => beginning
ignore_older => 0
}
}
Updated Input syntax (Read CSVs from S3)
input
{
s3
{
access_key_id => "XXXXXXX"
secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXXXXXXX"
bucket => "abc"
prefix => "allcsv/"
delete => false
interval => 60 # seconds
sincedb_path => '/dev/null'
}
}
Example
Path has value
C:/logstash-2.3.1/logstash-2.3.1/abc/allcsv/AppInfo_07112016 104337.csv
grok filter as below
grok
{
match => ["path", "%{GREEDYDATA:CName}/%{GREEDYDATA:FName}/%{GREEDYDATA:pathtype}_%{GREEDYDATA:pathlast}"]
}
Problem:
S3 dont have path object, I need path object because i have used grok filter into filter section and i need full file path (Couple of logics/Fields will be added at runtime).
How can i get path/filepath using S3 input ?
Is there a way to get path/filepath into Filter section ?
Is there any inbuild/local variable/object holds path value ?
Thanks for your help in advance,