The issue described here is related to CSV filter plugin with autodetect_column_names set.
I experience inconsistent behavior when processing CSV files with multiple workers.
Seems that the first line in the file is recognized as column names by the CSV filter by one of the workers and any following CSV lines which goes to other workers fail.
This makes sense to me as each worker has it's own instance of the filter and no global state is shared.
Please confirm this issue/limitation. Logstash has by default multi worker setup and there are no warnings in the CSV filter documentation of issues which would occur as a result.
input {
file {
path => "/input/**/*.csv"
}
}
filter {
csv {
autodetect_column_names => true
skip_header => true
}
...