# Multiple Files CSV automatic create index with filename and column header

**URL:** <https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235>\
**Category:** Logstash\
**Created:** [February 6, 2019, 6:54am UTC](https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235 "2019-02-06T06:54:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrewjo](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@andrewjo](https://discuss.elastic.co/u/andrewjo)\
**Post date:** [February 6, 2019, 6:54am UTC](https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235/1 "2019-02-06T06:54:32Z")

</div>

Hi Guys,

I have multiple CSV and I want to ingest into elasticsearch with logstash.  
Here is my logstash config :

```
file {
    path => '/tmp/email/*.csv'
    type => 'testcsv'
# start_position => 'beginning'
    sincedb_path => "/dev/null"
      }
}

filter {
 csv {
  separator => ","
  autodetect_column_names => "true"
 }
grok {
match => {
      "path" => "%{GREEDYDATA:filepath}/%{GREEDYDATA:filename}\_%{GREEDYDATA:filetime}\.csv"
      }
 }
  mutate {
gsub => [
  "filename", "[\s?\\?#-]", "."
]
lowercase => ["filename"]

 }
}

output {
   elasticsearch {
                hosts => ["localhost:9200"]
                sniffing => true
                manage_template => false
                index => "%{[filename]}-%{+YYYY.MM.dd}"
   }
}

```

My logstash config has successfully index first csv file.  
But the other file generate index with the same column name in first file.  
Please share tips to index for automatic create index with filename and column header.

Thanks before.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 6, 2019, 3:09pm UTC](https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235/2 "2019-02-06T15:09:04Z")

</div>

> [@andrewjo](#):
>
> But the other file generate index with the same column name in first file.  
> Please share tips to index for automatic create index with filename and column header.

If you have multiple CSV files with different sets of columns then a regular csv filter will not work. Once it has detected the column names they do not get reset when it starts processing the next file.

I think it would be possible to take the [code](https://github.com/logstash-plugins/logstash-filter-csv/blob/master/lib/logstash/filters/csv.rb) for the csv filter and change it so that it resets [columns](https://github.com/logstash-plugins/logstash-filter-csv/blob/5fbb285939b8b2cc47e2cf48a7029471dc95a848/lib/logstash/filters/csv.rb#L135) every time it sees a new value for the path field on the event.

---

<div class="post-metadata">

**Author:** ![andrewjo](https://avatars.discourse-cdn.com/v4/letter/a/b782af/32.png) [@andrewjo](https://discuss.elastic.co/u/andrewjo)\
**Post date:** [February 7, 2019, 4:44pm UTC](https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235/3 "2019-02-07T16:44:20Z")

</div>

Ok, Thanks Badger for your answer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 7, 2019, 4:46pm UTC](https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235/4 "2019-03-07T16:46:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.