# "delimiter" field behaving weirdly with \\n char for the line codec with UTF-16

**URL:** <https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341>\
**Category:** Logstash\
**Created:** [June 18, 2019, 7:48pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341 "2019-06-18T19:48:31Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 7:48pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/1 "2019-06-18T19:48:31Z")

</div>

Hi, I’m trying to parse UTF-16-encoded (with BOM), CRLF newlines CSV data from a TCP input, but I cant get it to be parsed correctly.

In my .conf file:

```auto
input {
	tcp {
		port => 5008
		type => "csv"
		codec => line {
			charset => "UTF-16LE"
			delimiter => "\r\n"
		}
	}
}

output {
	stdout{ codec => rubydebug }
}

```

When I import the data with `ncat localhost 5008 < mydata.csv`, I only get one giant event.

Thats where I started investigating. Reading the doc for the "delimiter" field ( [https://www.elastic.co/guide/en/logstash/7.1/plugins-codecs-line.html](https://www.elastic.co/guide/en/logstash/7.1/plugins-codecs-line.html) ), it says that

> Default value is `"\n"`

When I don’t set the delimiter field, I have other problems, but at least it separates events correctly. However, when I put `delimiter => "\n"` (which is the _default_ value, it shouldn’t change the output), **it parses as one big event.**

This leads me to think that there might be a problem with the way that the "delimiter" field for the line codec handles `\n` and/or `\r`. Is there something I might have overlooked?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 18, 2019, 7:55pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/2 "2019-06-18T19:55:13Z")

</div>

Unless you have config.support\_escapes enabled

```
delimiter => "\n"

```

is not the default

```
delimiter => "
"

```

is the default. You use a literal newline in the configuration file.

---

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 8:22pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/3 "2019-06-18T20:22:25Z")

</div>

That indeed seems to work for the `\n` when I enable this config.

However, it doesnt work with `\r\n`. I know that my lines are CRLF:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/7/1793abe70aa8438663b2caa08be7dd7304958e27.png)  
But I still get one big message. Any idea why?

Thanks.

---

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 8:30pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/4 "2019-06-18T20:30:11Z")

</div>

Using `UTF-16` instead of `UTF-16LE` fixed this problem. **EDIT: this is wrong, see my later messages.**

Now, messages are separated correctly, but the content does not seem to be understood by logstash: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7f76d0000efd8a4d2159aca3326e7249675061d.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 18, 2019, 8:38pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/5 "2019-06-18T20:38:03Z")

</div>

> [@res260](#):
>
> Any idea why?

Note that your data and your delimiters have opposite endianness. So if you get lines, all of your 16 bit characters have the bytes swapped. I believe it would be possible to fix that in a ruby filter.

---

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 8:44pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/6 "2019-06-18T20:44:48Z")

</div>

I see the same endianness, starts with 2 bytes representing the BOM, then little endian until 0D 00 0A 00, which is also little endian. Is that right?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 18, 2019, 8:46pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/7 "2019-06-18T20:46:12Z")

</div>

Yeah, I misread it.

---

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 8:46pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/8 "2019-06-18T20:46:17Z")

</div>

Also I made a mistake, changing to `UTF-16` did not fix the problem, I still cant get the `\r\n` delimiter to work.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 18, 2019, 8:49pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/9 "2019-06-18T20:49:09Z")

</div>

How are you setting the delimiter. And what platform (Windows vs. UNIX) are you on?

---

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 8:50pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/10 "2019-06-18T20:50:55Z")

</div>

Current input:

```auto
input {
	tcp {
		port => 5008
		type => "csv"
		codec => line {
			charset => "UTF-16"
			delimiter => "\r\n"
		}
	}
}

```

Current output: one big event. I enabled `config.support_escapes: true` in logstash.yml.

---

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 8:53pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/11 "2019-06-18T20:53:30Z")

</div>

I’m on Ubuntu 18.04. The file comes from a windows machine.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 18, 2019, 9:04pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/12 "2019-06-18T21:04:30Z")

</div>

Looking at the [code](https://github.com/logstash-plugins/logstash-codec-line/blob/d4734598a8c1e944f97ed1acafc03e3e1fc66feb/lib/logstash/codecs/line.rb#L32), the tokenizer knows nothing about the charset. So you need your delimiter to be '\r\0\n\0'.

There is a [PR](https://github.com/elastic/logstash/pull/9702) open to add support for \0, but at the moment I think you are out of luck. I do not know if @yaauie plans to merge this in a future version.

You could get rid of the codec and tokenize it yourself in a ruby filter.

---

<div class="post-metadata">

**Author:** ![res260](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/res260/32/48329_2.png) [@res260](https://discuss.elastic.co/u/res260)\
**Post date:** [June 18, 2019, 9:21pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/13 "2019-06-18T21:21:20Z")

</div>

Alright I’ll take a look at that. Many thanks for your time!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 16, 2019, 9:21pm UTC](https://discuss.elastic.co/t/delimiter-field-behaving-weirdly-with-n-char-for-the-line-codec-with-utf-16/186341/14 "2019-07-16T21:21:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.