# Logstash fails to auto generate column names from CSV file

**URL:** <https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265>\
**Category:** Logstash\
**Created:** [June 11, 2019, 6:54pm UTC](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265 "2019-06-11T18:54:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [June 11, 2019, 6:54pm UTC](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265/1 "2019-06-11T18:54:14Z")

</div>

Hi everyone,

I used the following Logstash configuration, but sometimes Logstash auto generates the column names from the CSV file, and sometimes its not (According to the pictures). Could you please clarify what is the issue here?

```
ruby {
    code => 'event.set("[@metadata][fields]", 1 + event.get("message").count(","))' 
}
if [@metadata][fields] > 2 {
    mutate {
        gsub => [
            "message", "[\"\"/\r]", ""
        ]
    }
    csv {
        separator => ","
        autodetect_column_names => true
        autogenerate_column_names => true
    }
    mutate {
        convert => {
            "percent_compressed" => "integer"
            "rosbagTimestamp" => "integer"
            "secs" => "integer"
            "status" => "integer"
            "stamp" => "integer"
            "goal_id" => "integer"
            "duration_time" => "float"
            "percent_uploaded" => "integer"
            "seq" => "integer"
            "nsecs" => "integer"
            # "total_space" => "integer"      
            # "free_space" => "integer"
            # "used_space" => "integer"   
        }
    }
}

```

**Without column names:**  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/f/1fd8b5cb75469c6aaa676c59ec4108a2ae98b9fa.png)

**With column names:**  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/6/86499daeaf43a7cb9b1bbcd3f88622d55bc238ed.png)

**Note:**

The JSON output pictures might not be the same. However, they come from the same CSV file.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 11, 2019, 7:36pm UTC](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265/2 "2019-06-11T19:36:29Z")

</div>

Do you have '--pipeline.workers 1' set? If not, then the row containing the headers may get sent to one worker thread, and another worker thread may set the column names from a data row.

---

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [June 11, 2019, 8:43pm UTC](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265/3 "2019-06-11T20:43:19Z")

</div>

@Badger, this is what I have in my pipeline.yml file,

```
- pipeline.id: pipeline1
  pipeline.workers: 1
  queue.type: persisted
  path.config: "/usr/share/logstash/pipeline/*.conf"

```

I'm not sure if the problem is the path.config because I set it as usr/share/logstash/pipeline, and I just checked and the logstash directory doesn't exist in that directory. I was using it when I create my docker image and transfer the files to /usr/share/logstash/... .

[Edit]:

I just changed path.config, and I'm still getting that issue.

```
- pipeline.id: pipeline1
  pipeline.workers: 1
  queue.type: persisted
  # path.config: "/usr/share/logstash/pipeline/*.conf"
  path.config: "/home/kourosh/Documents/metrics/pipeline/*.conf"
```

---

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [June 11, 2019, 9:06pm UTC](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265/4 "2019-06-11T21:06:06Z")

</div>

**Solution:**

According to @Badger, I had to change my pipeline.worker in logstash.yml file. Make sure to uncommet it if it is commented, and set it to pipeline.workers: 1.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 9, 2019, 9:06pm UTC](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265/5 "2019-07-09T21:06:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.