Logstash fails to auto generate column names from CSV file

Elastic Stack Logstash
1

Hi everyone,

I used the following Logstash configuration, but sometimes Logstash auto generates the column names from the CSV file, and sometimes its not (According to the pictures). Could you please clarify what is the issue here?

ruby {
    code => 'event.set("[@metadata][fields]", 1 + event.get("message").count(","))' 
}
if [@metadata][fields] > 2 {
    mutate {
        gsub => [
            "message", "[\"\"/\r]", ""
        ]
    }
    csv {
        separator => ","
        autodetect_column_names => true
        autogenerate_column_names => true
    }
    mutate {
        convert => {
            "percent_compressed" => "integer"
            "rosbagTimestamp" => "integer"
            "secs" => "integer"
            "status" => "integer"
            "stamp" => "integer"
            "goal_id" => "integer"
            "duration_time" => "float"
            "percent_uploaded" => "integer"
            "seq" => "integer"
            "nsecs" => "integer"
            # "total_space" => "integer"      
            # "free_space" => "integer"
            # "used_space" => "integer"   
        }
    }
}

Without column names:
image

With column names:
image

Note:

The JSON output pictures might not be the same. However, they come from the same CSV file.

2

Do you have '--pipeline.workers 1' set? If not, then the row containing the headers may get sent to one worker thread, and another worker thread may set the column names from a data row.

3

@Badger, this is what I have in my pipeline.yml file,

- pipeline.id: pipeline1
  pipeline.workers: 1
  queue.type: persisted
  path.config: "/usr/share/logstash/pipeline/*.conf"

I'm not sure if the problem is the path.config because I set it as usr/share/logstash/pipeline, and I just checked and the logstash directory doesn't exist in that directory. I was using it when I create my docker image and transfer the files to /usr/share/logstash/... .

[Edit]:

I just changed path.config, and I'm still getting that issue.

- pipeline.id: pipeline1
  pipeline.workers: 1
  queue.type: persisted
  # path.config: "/usr/share/logstash/pipeline/*.conf"
  path.config: "/home/kourosh/Documents/metrics/pipeline/*.conf"
4

Solution:

According to @Badger, I had to change my pipeline.worker in logstash.yml file. Make sure to uncommet it if it is commented, and set it to pipeline.workers: 1.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.