# Autodetect\_column\_name with 2 different CSV

**URL:** <https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310>\
**Category:** Logstash\
**Created:** [August 31, 2022, 5:09am UTC](https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310 "2022-08-31T05:09:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![EliottB](https://avatars.discourse-cdn.com/v4/letter/e/65b543/32.png) [@EliottB](https://discuss.elastic.co/u/EliottB)\
**Post date:** [August 31, 2022, 5:09am UTC](https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310/1 "2022-08-31T05:09:18Z")

</div>

Hello all,

I have an issue with the autodetect\_column\_name of my pipeline below:

```auto
filter {
  csv {
    separator =>","
    autodetect_column_names => true
      }

```

When I process the first CSV message  
country,city,name,address  
USA,NYC,John,TestStreet

It process correctly the message  
country =\> USA,  
city =\> NYC,  
name =\> John,  
address =\> TestStreet

But then when I process a second CSV  
country,city,address  
Canada,Vancouver,GreenStreet

It will mess up the message using the same columns detected from the first message  
country =\> Canada  
city =\> Vancouver  
name =\> GreenStreet

I already activated pipline.workers : 1 in my logstash.yml. I feel somehow it is buffering the first message columns it has detected and does not try to autodetect the column again for the new messages coming.

Thanks for your help !

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [August 31, 2022, 6:47am UTC](https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310/2 "2022-08-31T06:47:25Z")

</div>

Check [this](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265)

---

<div class="post-metadata">

**Author:** ![EliottB](https://avatars.discourse-cdn.com/v4/letter/e/65b543/32.png) [@EliottB](https://discuss.elastic.co/u/EliottB)\
**Post date:** [August 31, 2022, 7:42am UTC](https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310/3 "2022-08-31T07:42:34Z")

</div>

Hey Rios,

Thanks, I already set up the pipeline.workers to 1 and it is uncommented but it does not work. The ingesting keeps using the colums it detected the very first time and does not refresh at every new message.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 31, 2022, 4:41pm UTC](https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310/4 "2022-08-31T16:41:47Z")

</div>

With autodetect\_column\_names the [code](https://github.com/logstash-plugins/logstash-filter-csv/blob/30b2c232293f9b0e11bdc36938c32b15731de3e8/lib/logstash/filters/csv.rb#L143) eats the first line to use as column names and never changes them. See [this](https://discuss.elastic.co/t/using-ruby-to-count-the-number-of-fields-in-a-message-then-apply-csv-depending-on-the-count/177880/2) thread for a way to support multiple CSV formats.

---

<div class="post-metadata">

**Author:** ![EliottB](https://avatars.discourse-cdn.com/v4/letter/e/65b543/32.png) [@EliottB](https://discuss.elastic.co/u/EliottB)\
**Post date:** [September 1, 2022, 8:37am UTC](https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310/5 "2022-09-01T08:37:16Z")

</div>

Hey Badger,

Thanks a lot it works perfectly fine !

```auto
ruby { code => 'event.set("[@metadata][fields]", 1 + event.get("message").count(","))'}
if [@metadata][fields] == 3 {
 	  csv {
  		skip_header => true
  		separator =>","
  		columns => 
  			[
  			"column1",
  			"column2",
                        "column3"
                         ]
                   } }
else if [@metadata][fields] == 2 {
 	  csv {
  		skip_header => true
  		separator =>","
  		columns => 
  			[
  			"column1",
  			"column2"
                        ]
                    } }
else {
 	  csv {
  		skip_header => true
  		separator =>","
  		columns => 
  			[
  			"column1"
                        ]
                    } }
                   
               

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 29, 2022, 8:38am UTC](https://discuss.elastic.co/t/autodetect-column-name-with-2-different-csv/313310/6 "2022-09-29T08:38:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.