# CSV filter - How can I parse unquoted CSV? **URL:** \ **Category:** Logstash\ **Created:** [February 15, 2019, 11:01am UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566 "2019-02-15T11:01:31Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Skeeve](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@Skeeve](https://discuss.elastic.co/u/Skeeve)\ **Post date:** [February 15, 2019, 11:01am UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/1 "2019-02-15T11:01:31Z")
My Data looks similar to this: 2019-02-15T10:56:00.000+000\t{"test":"value","more":{"property":"value"}} but with a lot more fields. Issue here is that one field is a JSON structure which is unquoted. When I do not set "quote\_char" I get parse errors because there is "Illegal quoting in line 1.". When I set "quote\_char" to "" I get an errror because ":quote\_char has to be a single character String". So what's the correct way to parse CSV which does not have quotes? ---
**Author:** ![Skeeve](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@Skeeve](https://discuss.elastic.co/u/Skeeve)\ **Post date:** [February 15, 2019, 12:47pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/2 "2019-02-15T12:47:44Z")
My current approach is to use the "dissect" plugin: ``` filter { dissect { mapping => { message => "%{scan_id} %{result_id} %{created_scan} %{created_at} %{updated_at} %{updated_scan} %{scanner_type} %{duration} %{recurrentscan} %{freescan} %{dangerlevel} %{has_error} %{url} %{complete_request}" } } mutate { remove_field => ["message"] strip => ["complete_request"] convert => { "scan_id" => "integer" "result_id" => "integer" "scanner_type" => "string" "duration" => "integer" "recurrentscan" => "boolean" "freescan" => "boolean" "dangerlevel" => "integer" "has_error" => "boolean" "url" => "string" } } } ``` Unfortunately there seems to be no way here to tell logstash that 4 fields are date\_time, which I could do with the CSV plugin. So if there is anyone having an idea how to resolve this, feel free to add… ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 15, 2019, 2:50pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/3 "2019-02-15T14:50:32Z")
> [@Skeeve](#): > > Unfortunately there seems to be no way here to tell logstash that 4 fields are date\_time, which I could do with the CSV plugin. How do you do that using the csv filter? ---
**Author:** ![Skeeve](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@Skeeve](https://discuss.elastic.co/u/Skeeve)\ **Post date:** [February 15, 2019, 2:55pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/4 "2019-02-15T14:55:18Z")
> [@Badger](#): > > How do you do that using the csv filter? Does this help? [Csv filter plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-convert) > [@](#): > > Possible conversions are […] date\_time […] ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 15, 2019, 3:00pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/5 "2019-02-15T15:00:27Z")
You could use a date filter, or you could use mutate+add\_field to add a new field that comprises the four fields of interest, then use a csv field to chop that up and convert them. ---
**Author:** ![Skeeve](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@Skeeve](https://discuss.elastic.co/u/Skeeve)\ **Post date:** [February 15, 2019, 6:47pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/6 "2019-02-15T18:47:46Z")
Sounds a bit overcomplicated. I still wonder why it's not possible to parse unquoted CSV. ---
**Author:** ![Skeeve](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@Skeeve](https://discuss.elastic.co/u/Skeeve)\ **Post date:** [February 15, 2019, 9:42pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/7 "2019-02-15T21:42:25Z")
After digging a bit deeper I found that the CSV filter is based on ruby's CSV class which in turn follows RFC 4180 which states that "If fields are not enclosed with double quotes, then double quotes may not appear inside the fields." -\> So here was my wrong assumption. MySQL (which generates my data) does not quote fields at all. So the only way the CSV filter might be able to parse MySQL's CSV is to set ``` :liberal_parsing ``` to true. Is there any way I could do that? Oh… And that's only available since CSV 2.4.0 (feature #11839) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 15, 2019, 10:05pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/8 "2019-02-15T22:05:55Z")
> [@Skeeve](#): > > Is there any way I could do that? No, the filter does not support that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 15, 2019, 10:05pm UTC](https://discuss.elastic.co/t/csv-filter-how-can-i-parse-unquoted-csv/168566/9 "2019-03-15T22:05:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.