Error reading empty line from CSV file with CSV codec

My input CSV files will have empty lines interspersed in them. The CSV codec decoder generates the following error when encountering an empty line:

[ERROR][filewatch.tailmode.handlers.grow][main][62dd5eb2b6763ff5522ed5480a076813eac7068008e52a3c7908c2befa19642e] controlled_read general error reading {:path=>"/opt/logstash/test/data/status-monitoring-hyosung-proto-TEST.csv", :exception=>NoMethodError, :message=>"undefined method `each_with_index' for nil:NilClass", :backtrace=>

After many attempts to get the configuration right, I stripped the config down to the bare minimum, but get the same error. Here's the stripped down, minimal config:

input {
  file {
    mode => "tail"
    path => ["/opt/logstash/test/data/status-monitoring-hyosung-proto-TEST.csv"]
    start_position => "beginning"
    sincedb_path => ["/tmp/logstash-output-jdbc-proto-sincedb"]
    codec => csv {
    }
  }
}
output {
  stdout {}
}

If I remove the csv codec stanza from the config, logstash ingests the csv file without issue. Here's the stripped down csv file I'm using for troubleshooting:

date,time,terminalID
2024-01-21,00:00:05,MP002001

2024-01-21,00:00:14,MP002002

2024-01-21,00:00:16,MP002003

Is it a known issue that the csv codec decoder cannot handle empty lines, or am I missing some configuration setting for dealing with empty lines?

The codec cannot handle empty rows. The filter has a skip_empty_rows option that tags blank lines. So you use a csv filter instead of a csv codec.

Switching to a csv filter was exactly what I needed.
Thank you, Badger.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.