Logstash S3 input + Looking Path object


(sarthak) #1

Hello All,

I have replace my old input with Amazon S3 input because i Need to read CSVs from S3 bucket instead of my local drive.

Old Input syntax (Read CSVs from my local)

input
{
file
{
sincedb_path => '/dev/null'
path => "C:/logstash-2.3.1/logstash-2.3.1/abc/allcsv/*.csv"
start_position => beginning
ignore_older => 0
}
}

Updated Input syntax (Read CSVs from S3)

input
{

s3
{
  access_key_id => "XXXXXXX"
  secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  bucket => "abc"
  prefix => "allcsv/"
  delete => false
  interval => 60 # seconds
  sincedb_path => '/dev/null'
  
}

}

Example

Path has value
C:/logstash-2.3.1/logstash-2.3.1/abc/allcsv/AppInfo_07112016 104337.csv

grok filter as below
grok
{
match => ["path", "%{GREEDYDATA:CName}/%{GREEDYDATA:FName}/%{GREEDYDATA:pathtype}_%{GREEDYDATA:pathlast}"]
}

Problem:
S3 dont have path object, I need path object because i have used grok filter into filter section and i need full file path (Couple of logics/Fields will be added at runtime).
How can i get path/filepath using S3 input ?
Is there a way to get path/filepath into Filter section ?
Is there any inbuild/local variable/object holds path value ?

Thanks for your help in advance,


(J Meter Dude) #2

Were you able to solve your issue?


(sarthak) #3

Yes, I updated S3.rb into Logstassh folder,