# Encoding utf-8 doesn't honor BOM

**URL:** <https://discuss.elastic.co/t/encoding-utf-8-doesnt-honor-bom/46671>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 7, 2016, 12:13pm UTC](https://discuss.elastic.co/t/encoding-utf-8-doesnt-honor-bom/46671 "2016-04-07T12:13:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![prehor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prehor/32/8977_2.png) [@prehor](https://discuss.elastic.co/u/prehor)\
**Post date:** [April 7, 2016, 12:13pm UTC](https://discuss.elastic.co/t/encoding-utf-8-doesnt-honor-bom/46671/1 "2016-04-07T12:13:54Z")

</div>

Hi,

Windows often prepend UTF-8 BOM to text files, which is legal - see [Can a UTF-8 data stream contain the BOM character (in UTF-8 form)?](http://unicode.org/faq/utf_bom.html#bom5). In my case it happens in Exchange 2010 Message Tracking logs.

Filebeat must strip UTF-8 BOM from files with UTF-8 encoding which it doesn't do and BOM appears in message field for the first line in the file:

```auto
{"@timestamp":"2016-04-07T11:58:36.922Z","beat":{"hostname":"XXXXX","name":"XXXXX"},"count":1,"fields":null,"input_type":"log","message":"<U+FEFF>#Software: Microsoft Exchange Server","offset":0,"source":"exchange/MSGTRK20160405-1.LOG","type":"exchange"}

```

Filebeat config:

filebeat:  
prospectors:  
-  
document\_type: exchange  
input\_type: log  
paths:  
- exchange/MSGTRK2\*.LOG  
encoding: utf-8

output:  
file:  
path: logstash/output  
name: exchange

Hex dump of first line in file:

```
00000000 ef bb bf 23 53 6f 66 74 77 61 72 65 3a 20 4d 69 |...#Software: Mi|
00000010 63 72 6f 73 6f 66 74 20 45 78 63 68 61 6e 67 65 |crosoft Exchange|
00000020 20 53 65 72 76 65 72 0d 0a | Server..|

```

First three bytes EF BB BF (UTF-8 encoded BOM) are decoded to unicode character FE FF (which is used as BOM for UTF-16 encoding) and it appears in message field.

UTF-8 BOM could be used to detect UTF-8 encoding.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 7, 2016, 12:33pm UTC](https://discuss.elastic.co/t/encoding-utf-8-doesnt-honor-bom/46671/2 "2016-04-07T12:33:56Z")

</div>

we do not try to detect whatever encoding (besides the `utf16-bom` codec) we're dealing with, as BOM is used very rarely. Still BOM should be removed from events. Please report this [as bug on github](https://github.com/elastic/beats/issues). Maybe fix includes 'just' updating library used for decoding different character-sets.

---

<div class="post-metadata">

**Author:** ![prehor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prehor/32/8977_2.png) [@prehor](https://discuss.elastic.co/u/prehor)\
**Post date:** [April 7, 2016, 1:11pm UTC](https://discuss.elastic.co/t/encoding-utf-8-doesnt-honor-bom/46671/3 "2016-04-07T13:11:33Z")

</div>

OK, I filled bug report [#1349](https://github.com/elastic/beats/issues/1349).

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 7, 2016, 4:19pm UTC](https://discuss.elastic.co/t/encoding-utf-8-doesnt-honor-bom/46671/4 "2016-04-07T16:19:56Z")

</div>

thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:53pm UTC](https://discuss.elastic.co/t/encoding-utf-8-doesnt-honor-bom/46671/5 "2017-07-05T21:53:39Z")

</div>