# Csv filter plugin: how to deal with masked quotations?

**URL:** https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488
**Category:** Logstash
**Created:** [August 23, 2019, 10:10am UTC](https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488 "2019-08-23T10:10:46Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)
#### Post date: [August 23, 2019, 10:10am UTC](https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488/1 "2019-08-23T10:10:46Z")

</div>

Hi,

I have following csv file.

```
"@timestamp";"col_a";"col_b";"col_c";"col_d"
"2019-08-23 04:43:16.821";"<?xml version=\"1.0\" encoding=\"UTF-8\"?>";b;c;d

```

I am trying to parse it in logstash with following filter:

```
filter
{
  csv
  {
    autodetect_column_names => true
    autogenerate_column_names => true
    separator => ";"
    source => "message"
    skip_empty_columns => "true"
    target=> "mycsv"
  }
}

```

2nd line is throwing following error:

```
"2019-08-23 04:43:16.821";"<?xml version=\"1.0\" encoding=\"UTF-8\"?>";b;c;d
[2019-08-23T12:02:38,204][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"\"2019-08-23 04:43:16.821\";\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\";b;c;d\r", :exception=>#<CSV::MalformedCSVError: Missing or stray quote in line 1>}
[2019-08-23T12:02:38,208][INFO][logstash.outputs.file] Opening file {:path=>"c:/work/elastic/input/myoutput.json"}
[2019-08-23T12:02:38,216][INFO][logstash.outputs.file] Opening file {:path=>"c:/work/elastic/input/myoutput.log"}
{
    "@timestamp" => 2019-08-23T10:02:38.099Z,
          "host" => "dtpbl0319",
      "@version" => "1",
       "message" => "\"2019-08-23 04:43:16.821\";\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\";b;c;d\r",
          "tags" => [
        [0] "_csvparsefailure"
    ]
}

```

Changing the `quote_char` is not an option to me. Input is coming from logs where the code can contain single or double quotes.

Is there a way csv filter can deal with masked quote\_chars, so that it handles them as normal text?

Background: I am exporting results of queries via kibana's csv export from customer. Now I am trying to import the results back to an independant elasticsearch instance for our developers.

Thanks, Andreas

---

<div class="post-metadata">

### Author: ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)
#### Post date: [August 23, 2019, 10:38am UTC](https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488/2 "2019-08-23T10:38:01Z")

</div>

OK, sorry, I found the issue here:

> <https://github.com/logstash-plugins/logstash-filter-csv/issues/64>

kibana csv output is masking the doublequotes `"` correctly as`""`  
That is accepted by csv filter plugin.

I changed it to " during debugging. - My fault.

The main cause of my issue was, that kibanas csv export is not masking newlines with \n. So one csv row may consist of many lines if you have multiline values. But csv input plugin does not like it. Thought first it was an issue with the quotes.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 23, 2019, 2:05pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488/3 "2019-08-23T14:05:03Z")

</div>

If the data really does look like that then indeed the csv filter will choke on it. I would work around it using

```
 mutate { gsub => ["message", '[\\]"', "!!BackslashDoubleQuote!!" ] }

```

Then gsub back after the csv.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 20, 2019, 2:05pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-how-to-deal-with-masked-quotations/196488/4 "2019-09-20T14:05:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
