Multiple workers fail to process CSV files when auto detect column names is set

Elastic Stack Logstash
1

The issue described here is related to CSV filter plugin with autodetect_column_names set.
I experience inconsistent behavior when processing CSV files with multiple workers.
Seems that the first line in the file is recognized as column names by the CSV filter by one of the workers and any following CSV lines which goes to other workers fail.
This makes sense to me as each worker has it's own instance of the filter and no global state is shared.

Please confirm this issue/limitation. Logstash has by default multi worker setup and there are no warnings in the CSV filter documentation of issues which would occur as a result.

input {
  file {
    path => "/input/**/*.csv"    
  }
}

filter {
  
  csv {
      autodetect_column_names => true
      skip_header => true
  } 
...
2

Not so. The documentation for that option says " Logstash pipeline workers must be set to 1 for this option to work".

3

Missed that,
Thank you

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.