Malformed csv error while reading csv file with Logstash

Elastic Stack Logstash
1

Hi all,

I try to read a .csv file using logstash, it works like a charm but for just a few lines I get the error "(MalformedCSVError) Illegal quoting in line 1."

The error occurs whenever a field contains a value with a " as value inside of it.

The inputted .csv line has the following structure:
field1;field2;"field3 with ""quoted"" text";field4;

So whenever a special character is kept in a field it get enclosed in a double quote. Whenever a double qoute is kept in that same field, it will be preceeded by antoher double quote like the csv convention describes. Still I get the malformed csv error from Logstash, what is the reason the error gets thrown and how can I fix this?

Thanks in advance,
Rick

2

You can try to use a gsub filter to replace all double quoted in the message with a single quote, so you would end up with something like this:

field1;field2;"field3 with 'quoted' text";field4;

This may work:

mutate {
    gsub => ["message",'""',"'"]
}