Error reading empty line from CSV file with CSV codec

Elastic Stack Logstash
1

My input CSV files will have empty lines interspersed in them. The CSV codec decoder generates the following error when encountering an empty line:

[ERROR][filewatch.tailmode.handlers.grow][main][62dd5eb2b6763ff5522ed5480a076813eac7068008e52a3c7908c2befa19642e] controlled_read general error reading {:path=>"/opt/logstash/test/data/status-monitoring-hyosung-proto-TEST.csv", :exception=>NoMethodError, :message=>"undefined method `each_with_index' for nil:NilClass", :backtrace=>

After many attempts to get the configuration right, I stripped the config down to the bare minimum, but get the same error. Here's the stripped down, minimal config:

input {
  file {
    mode => "tail"
    path => ["/opt/logstash/test/data/status-monitoring-hyosung-proto-TEST.csv"]
    start_position => "beginning"
    sincedb_path => ["/tmp/logstash-output-jdbc-proto-sincedb"]
    codec => csv {
    }
  }
}
output {
  stdout {}
}

If I remove the csv codec stanza from the config, logstash ingests the csv file without issue. Here's the stripped down csv file I'm using for troubleshooting:

date,time,terminalID
2024-01-21,00:00:05,MP002001

2024-01-21,00:00:14,MP002002

2024-01-21,00:00:16,MP002003

Is it a known issue that the csv codec decoder cannot handle empty lines, or am I missing some configuration setting for dealing with empty lines?

2

The codec cannot handle empty rows. The filter has a skip_empty_rows option that tags blank lines. So you use a csv filter instead of a csv codec.

3

Switching to a csv filter was exactly what I needed.
Thank you, Badger.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.