# Logstash CSV Filter Using Unicode delimiter ( SOH )

**URL:** <https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517>\
**Category:** Logstash\
**Created:** [October 19, 2015, 6:32pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517 "2015-10-19T18:32:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![samnik60](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@samnik60](https://discuss.elastic.co/u/samnik60)\
**Post date:** [October 19, 2015, 6:32pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517/1 "2015-10-19T18:32:04Z")

</div>

HI guys,  
I am trying to parse a CSV File which has a delimiter of (SOH) i.e Hexa value \x01 ( unicode value \u0001 ), but logstash csv filter seems to ignore the seperator mentioned .  
example

filter{  
csv{  
separator =\> "\u0001"  
}  
}

Does logstash csv filter support unicode characters or special characters as delimiter???

Thanks,  
sam

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 19, 2015, 6:35pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517/2 "2015-10-19T18:35:46Z")

</div>

The [string type documentation](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) doesn't mention anything about this so one should assume escape sequences are unsupported. See related thread below:

> [@Ctrl A separated csv file elasticsearch not indexing using logstash](https://discuss.elastic.co/t/ctrl-a-separated-csv-file-elasticsearch-not-indexing-using-logstash/32077):
>
> 0 down vote favorite I am using the config: input { file { path =\> "/tmp/test12.csv" type =\> "Sample" start\_position =\> "beginning" } } filter { csv { columns =\> ["col1","col2,"col3","col4"] separator =\> '\u0001' } } output { elasticsearch { action =\> "index" host =\> "localhost" index =\> "t2" document\_id =\> "%{col1}" } } For separator using '\u0001' but its not working

---

<div class="post-metadata">

**Author:** ![samnik60](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@samnik60](https://discuss.elastic.co/u/samnik60)\
**Post date:** [October 19, 2015, 8:00pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517/3 "2015-10-19T20:00:33Z")

</div>

Hi magnus,  
is this a feature not present in logstash csv filter or a fundamental limitation that logstash cannot support this as it doesnt allow unicode characters in its string data type.

Thanks,  
sam

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 19, 2015, 8:14pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517/4 "2015-10-19T20:14:08Z")

</div>

Logstash strings do not support escape sequences as a way to represent non-printable characters. If you can't put a literal \u0001 in the file (which the topic I linked to indicated didn't work) you're out of luck. This has nothing to do with the csv filter.

---

<div class="post-metadata">

**Author:** ![samnik60](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@samnik60](https://discuss.elastic.co/u/samnik60)\
**Post date:** [October 20, 2015, 12:34pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517/5 "2015-10-20T12:34:05Z")

</div>

I worked around this issue, i used a mutate filter as given below to replace the unicode delimiter to ascii then parse using csv

```
    mutate{
        gsub => ["message","\u0001",","]
    }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:26am UTC](https://discuss.elastic.co/t/logstash-csv-filter-using-unicode-delimiter-soh/32517/6 "2017-07-06T05:26:02Z")

</div>