# \[bug\] Syslog RFC 5424 parser does not properly handle escaped characters in structured data

**URL:** <https://discuss.elastic.co/t/bug-syslog-rfc-5424-parser-does-not-properly-handle-escaped-characters-in-structured-data/364483>\
**Category:** Elastic Agent\
**Tags:** libbeat, integrations\
**Created:** [August 6, 2024, 2:40pm UTC](https://discuss.elastic.co/t/bug-syslog-rfc-5424-parser-does-not-properly-handle-escaped-characters-in-structured-data/364483 "2024-08-06T14:40:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pcollardez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pcollardez/32/136577_2.png) [@pcollardez](https://discuss.elastic.co/u/pcollardez)\
**Post date:** [August 6, 2024, 2:40pm UTC](https://discuss.elastic.co/t/bug-syslog-rfc-5424-parser-does-not-properly-handle-escaped-characters-in-structured-data/364483/1 "2024-08-06T14:40:54Z")

</div>

Hello,

I'm trying to collect logs from different appliances by using an Elastic Agent 8.14.3 with the Custom UDP Logs integration 1.19.1.

One of the appliance is sending RFC 5424 formatted logs, with a structured data part and a regular part.

The structured data has values containing escaped "]" and this seems to trigger a bug in the parser if the escaped "]" is not the last character of a value.

For example, the following message is properly parsed:

```auto
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011" somekey="[value\]"][examplePriority@32473 class="high"] Some message

```

and the extracted structured data is as expected:

```auto
[exampleSDID@32473 iut="3" eventSource="Application" eventID="1011" somekey="[value\]"][examplePriority@32473 class="high"]

```

But this message is not:

```auto
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011" somekey="[value\] more data"][examplePriority@32473 class="high"] Some message

```

for which the extracted structured data is:

```auto
more data"][examplePriority@32473 class="high"]

```

Note: The messages above can be directly tested from the test unit for the rfc5425 parser of the syslog processor in libbeat

[https://github.com/elastic/beats/blob/main/libbeat/reader/syslog/rfc5424\_test.go](https://github.com/elastic/beats/blob/main/libbeat/reader/syslog/rfc5424_test.go)

---

<div class="post-metadata">

**Author:** ![taylor-swanson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taylor-swanson/32/100638_2.png) [@taylor-swanson](https://discuss.elastic.co/u/taylor-swanson)\
**Post date:** [August 6, 2024, 5:36pm UTC](https://discuss.elastic.co/t/bug-syslog-rfc-5424-parser-does-not-properly-handle-escaped-characters-in-structured-data/364483/2 "2024-08-06T17:36:50Z")

</div>

Hi @pcollardez,

Thank you for this report. This is indeed a bug with the parser. I've written a public issue for it over on Github: [Syslog reader/processor does not handle escaped brackets in structured data fields · Issue #40445 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/40445)

---

<div class="post-metadata">

**Author:** ![pcollardez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pcollardez/32/136577_2.png) [@pcollardez](https://discuss.elastic.co/u/pcollardez)\
**Post date:** [August 6, 2024, 9:23pm UTC](https://discuss.elastic.co/t/bug-syslog-rfc-5424-parser-does-not-properly-handle-escaped-characters-in-structured-data/364483/3 "2024-08-06T21:23:45Z")

</div>

There is also a bug if the regular message contains a "]":

```auto
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011" somekey="value"][examplePriority@32473 class="high"] Some message [value] more data

```

In this case, the extracted "structured" data is:

```auto
Some message [value]

```

and the regular message:

```auto
more data

```

instead of the expected ones.

On another level, it seems that the UTF-8 BOM (\xEFBBBF) is not processed if present at the beginning of the regular message:

```auto
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011" somekey="value"][examplePriority@32473 class="high"] \xEFBBBFSome message

```

but that would only forward the BOM in the value of the field, which is not really a bug or a real problem as long as there is no decoding attempt.

---

<div class="post-metadata">

**Author:** ![taylor-swanson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taylor-swanson/32/100638_2.png) [@taylor-swanson](https://discuss.elastic.co/u/taylor-swanson)\
**Post date:** [August 7, 2024, 1:53pm UTC](https://discuss.elastic.co/t/bug-syslog-rfc-5424-parser-does-not-properly-handle-escaped-characters-in-structured-data/364483/4 "2024-08-07T13:53:09Z")

</div>

I've got a fix in place for ']' in the regular message as well.

Regarding the BOM, is that even the correct escape sequence? To me that just looks like 'EF' hex, followed by a regular "BBBF". We have tests in place already for the BOM and I can't reproduce the issue. We use " \ufeff" for the escape sequence in tests and the live code. This is also what is used by Go: [text/encoding/unicode/override.go at master · golang/text · GitHub](https://github.com/golang/text/blob/master/encoding/unicode/override.go#L52)

EDIT: For a hex escape sequence, I would expect to see something like this: " \xEF\xBB\xBF". I tested with this in our unit tests and it is not included in the final message as expected.

---

<div class="post-metadata">

**Author:** ![pcollardez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pcollardez/32/136577_2.png) [@pcollardez](https://discuss.elastic.co/u/pcollardez)\
**Post date:** [August 7, 2024, 2:16pm UTC](https://discuss.elastic.co/t/bug-syslog-rfc-5424-parser-does-not-properly-handle-escaped-characters-in-structured-data/364483/5 "2024-08-07T14:16:16Z")

</div>

Thank you for the fixes about the brackets.

About the BOM, you're right, I made a mistake and you got it : I meant \xEF\xBB\xBF.