# Logstash can't recognize csv columns

**URL:** <https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396>\
**Category:** Logstash\
**Created:** [August 22, 2019, 7:17pm UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396 "2019-08-22T19:17:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![m.thecoder](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@m.thecoder](https://discuss.elastic.co/u/m.thecoder)\
**Post date:** [August 22, 2019, 7:17pm UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396/1 "2019-08-22T19:17:42Z")

</div>

Hello Guys,

I've configured logstah to read multiple csv files, each one has a different columns set (different column names), it reads the first one successfully, but it never reads the columns for any csv after reading the first one!

here is my code:

```auto
input {
    file {
       path => '/tmp/*.csv'
       type => 'testcsv'
       sincedb_path => "/dev/null"
    }
}

filter {
 csv {
  separator => ","
  autodetect_column_names => "true"
 }
}

output {
   elasticsearch {
                hosts => ["localhost:9200"]
                index => "%{tmp_index}"
   }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 22, 2019, 8:12pm UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396/2 "2019-08-22T20:12:35Z")

</div>

autodetect\_column\_names uses the very first event that logstash sees to determine the column names and never changes them. You could [count columns](https://discuss.elastic.co/t/using-ruby-to-count-the-number-of-fields-in-a-message-then-apply-csv-depending-on-the-count/177880/2) or else do a conditional based on the filename.

---

<div class="post-metadata">

**Author:** ![m.thecoder](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@m.thecoder](https://discuss.elastic.co/u/m.thecoder)\
**Post date:** [August 22, 2019, 8:19pm UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396/3 "2019-08-22T20:19:10Z")

</div>

> [@Badger](#):
>
> You could [count columns](https://discuss.elastic.co/t/using-ruby-to-count-the-number-of-fields-in-a-message-then-apply-csv-depending-on-the-count/177880/2) or else do a conditional based on the filename

Thanks for the reply, but what would I do with the fields number or the conditional on having the file's name changed in the path field?  
Sorry, I didn't catch your drift!

```auto
ruby { code => 'event.set("[@metadata][fields]", 1 + event.get("message").count(","))' }

```

because the listener might have files at anytime and with same # of columns but different names, and I can't have a hard-code **if else** condition, like the one in the link you've put in the reply.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 22, 2019, 8:34pm UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396/4 "2019-08-22T20:34:04Z")

</div>

If you want to ingest files with different formats (different sets of columns) the processing has to be conditional on which format a given event has. If you do not want to use if/else then ingest files one at a time.

---

<div class="post-metadata">

**Author:** ![m.thecoder](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@m.thecoder](https://discuss.elastic.co/u/m.thecoder)\
**Post date:** [August 22, 2019, 8:56pm UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396/5 "2019-08-22T20:56:47Z")

</div>

> [@Badger](#):
>
> If you want to ingest files with different formats (different sets of columns) the processing has to be conditional on which format a given event has

can you give me an example about this part? because the files won't be ingested at the same time, I might have one file to ingest now and three files after X minutes and so on.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 23, 2019, 12:01am UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396/6 "2019-08-23T00:01:35Z")

</div>

> [@m.thecoder](#):
>
> can you give me an example about this part?

That's the post I linked to. In that case the conditional was based on the number of columns, but it could also be based on a regexp match to the filename.

If you want to use autodetect\_column\_names then you would have to run logstash one time for each file, which is going to be very expensive for small files (it takes logstash about 45 seconds of CPU to start up on my system).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 20, 2019, 12:01am UTC](https://discuss.elastic.co/t/logstash-cant-recognize-csv-columns/196396/7 "2019-09-20T00:01:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.