Logstash failing for csv load with double quotes

Elastic Stack Logstash
1

Am using csv filter pluggin with separator => '|', it works if we remove the quotes for "QA". Ay work around to make this work or is this limitation with CSV filter.

Here is the sample load data:

QA8_COMMON|1044|AC|101|AUTOMATION-FDX1044NEWACCT|FDx "QA" Test Account@ &1044|TEST123|11|11|-1|606501|359006|0| QA8_COMMON|1608991|AC|2000|AUTOMATION-FDX1044NEWACCT|FDx "QA" Test Account@ &1044||11|11||875247|410628|176593|141596

Logastash error :

[2019-12-02T10:31:19,273][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"QA8_COMMON|1608991|AC|2000|AUTOMATION-FDX1044NEWACCT|FDx "QA" $ Test Account@ &1044||11|11||875247|410628|176593|141596", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

[2019-12-02T10:31:19,291][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"QA8_COMMON|1044|AC|101|AUTOMATION-FDX1044NEWACCT|FDx "QA" $ Test Account@ &1044|TEST123|11|11|-1|606501|359006|0|", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

2

The csv filter requires that the entire field be in quotes if there are quotes in it. A workaround would be to change the quote_char option to something does not occur in the line.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.