Logstash fails to auto generate column names from CSV file

[EZprogramming (Kourosh Azizi)]

Hi everyone,

I used the following Logstash configuration, but sometimes Logstash auto generates the column names from the CSV file, and sometimes its not (According to the pictures). Could you please clarify what is the issue here?

ruby {
    code => 'event.set("[@metadata][fields]", 1 + event.get("message").count(","))' 
}
if [@metadata][fields] > 2 {
    mutate {
        gsub => [
            "message", "[\"\"/\r]", ""
        ]
    }
    csv {
        separator => ","
        autodetect_column_names => true
        autogenerate_column_names => true
    }
    mutate {
        convert => {
            "percent_compressed" => "integer"
            "rosbagTimestamp" => "integer"
            "secs" => "integer"
            "status" => "integer"
            "stamp" => "integer"
            "goal_id" => "integer"
            "duration_time" => "float"
            "percent_uploaded" => "integer"
            "seq" => "integer"
            "nsecs" => "integer"
            # "total_space" => "integer"      
            # "free_space" => "integer"
            # "used_space" => "integer"   
        }
    }
}

Without column names:

With column names:

Note:

The JSON output pictures might not be the same. However, they come from the same CSV file.

[Badger]

Do you have '--pipeline.workers 1' set? If not, then the row containing the headers may get sent to one worker thread, and another worker thread may set the column names from a data row.

[EZprogramming (Kourosh Azizi)]

@Badger, this is what I have in my pipeline.yml file,

- pipeline.id: pipeline1
  pipeline.workers: 1
  queue.type: persisted
  path.config: "/usr/share/logstash/pipeline/*.conf"

I'm not sure if the problem is the path.config because I set it as usr/share/logstash/pipeline, and I just checked and the logstash directory doesn't exist in that directory. I was using it when I create my docker image and transfer the files to /usr/share/logstash/... .

[Edit]:

I just changed path.config, and I'm still getting that issue.

- pipeline.id: pipeline1
  pipeline.workers: 1
  queue.type: persisted
  # path.config: "/usr/share/logstash/pipeline/*.conf"
  path.config: "/home/kourosh/Documents/metrics/pipeline/*.conf"

[EZprogramming (Kourosh Azizi)]

Solution:

According to @Badger, I had to change my pipeline.worker in logstash.yml file. Make sure to uncommet it if it is commented, and set it to pipeline.workers: 1.