Logstash.filters.csv :exception=>#<RuntimeError: Invalid FieldReference: `[]`>}

[Zeal0us]

Getting lines like this:

[2019-04-08T08:13:50,464][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"", :exception=>#<RuntimeError: Invalid FieldReference: []>}

when trying to parse some csv files.

My input and filter look like:

input {
  file {
    path => ["/etc/logstash/csv_files/*.some.domain.net/*.csv"]
    mode => "read"
    ignore_older => 86400
  }
}
filter {
    csv {
        autodetect_column_names => true
        skip_empty_columns => true
        skip_header => true
    }

}

Any thoughts on why I may be getting these errors?

[Zeal0us]

It appears FileReference.java does not like 1 item arrays, and thinks they are field references. Just used a string for a path and that seemed to fix it.

I did have to clear out all the queued data in (for me) /var/lib/logstash/queue/ while the process was stopped, then restart it.

Make sure you also have pipeline workers set to 1, or the other processes will interpret rows other than the first as headers.

[Badger]

@Zeal0us

field=>"message", :source=>""

That suggests that there is a blank line in the file, but I would expect a different exception for a blank line.

[Zeal0us]

I replaced the source with "" and that seemed to cause it to not render on the page. That said, I posted how I was able to solve it.

[Badger]

@Zeal0us

Just used a string for a path and that seemed to fix it.

It is extremely unlikely that changing that could prevent the csv filter throwing an exception.

[Zeal0us]

Odd, since it did seem to fix it.
Note that the error seems to be thrown by FileReference.java (see line 283) where it looks for ambiguous syntax. It took me a very long time to track it down specifically because, as you mention, it makes little sense that it would be the CSV plugin throwing the error, but here we are.

I believe it's an issue of parsing one item arrays as fields.