Multiple Files CSV automatic create index with filename and column header

Elastic Stack Logstash
1

Hi Guys,

I have multiple CSV and I want to ingest into elasticsearch with logstash.
Here is my logstash config :

file {
    path => '/tmp/email/*.csv'
    type => 'testcsv'
#   start_position => 'beginning'
    sincedb_path => "/dev/null"
      }
}

filter {
 csv {
  separator => ","
  autodetect_column_names => "true"
 }
grok {
match => {
      "path" => "%{GREEDYDATA:filepath}/%{GREEDYDATA:filename}\_%{GREEDYDATA:filetime}\.csv"
      }
 }
  mutate {
gsub => [
  "filename", "[\s?\\?#-]", "."
]
lowercase => [ "filename"]

 }
}

output {
   elasticsearch {
                hosts => ["localhost:9200"]
                sniffing => true
                manage_template => false
                index => "%{[filename]}-%{+YYYY.MM.dd}"
   }
}

My logstash config has successfully index first csv file.
But the other file generate index with the same column name in first file.
Please share tips to index for automatic create index with filename and column header.

Thanks before.

2

If you have multiple CSV files with different sets of columns then a regular csv filter will not work. Once it has detected the column names they do not get reset when it starts processing the next file.

I think it would be possible to take the code for the csv filter and change it so that it resets columns every time it sees a new value for the path field on the event.

3

Ok, Thanks Badger for your answer.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.