# Can't filter Unicode logs at all **URL:** \ **Category:** Logstash\ **Created:** [August 30, 2018, 12:49pm UTC](https://discuss.elastic.co/t/cant-filter-unicode-logs-at-all/146701 "2018-08-30T12:49:45Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![yolt](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@yolt](https://discuss.elastic.co/u/yolt)\ **Post date:** [August 30, 2018, 12:49pm UTC](https://discuss.elastic.co/t/cant-filter-unicode-logs-at-all/146701/1 "2018-08-30T12:49:45Z")
Hello, I need help to find out how to set **Filebeat** or **Logstash** or both... to be able successfully filter logs by message from files which have set encoding as **Unicode** I will provide screenshots to describe problem as best as i can. Currently i'm using 6.4.0 version for Filebeat, Logstash, Elasticsearch and Kibana. **Logstash** , **Elasticsearch** and **Kibana** are runing as docker services on one NODE. - Docker version 17.12.1-ce - OS linux Ubuntu 16.04 LTS **Filebeat** is harvesting logs on Windows 10 as process and sending them to Logstash. Here is my config for each utility. **Filebeat** filebeat.yml ``` filebeat.inputs: - type: log paths: - C:\Users\C5260750\Desktop\customlogs\* fields: level: debug status of machine: running review: 1 multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after output.logstash: hosts: ["10.55.177.60:5044"] ``` **Logstash** logstash.config ``` input { beats { port => 5044 } } output { elasticsearch { hosts => "http://10.55.177.60:9200" user => elastic password => changeme } stdout { codec => rubydebug } } ``` logstash.yml ``` http.host: "0.0.0.0" path.config: /usr/share/logstash/pipeline xpack.monitoring.elasticsearch.url: [http://10.55.177.60:9200](http://10.55.177.60:9200/) xpack.monitoring.elasticsearch.username: elastic xpack.monitoring.elasticsearch.password: changeme ``` **Elasticsearch** ``` Default ``` **Kibana** kibana.yml ``` server.name: kibana server.host: "0" elasticsearch.url: http://10.55.177.60:9200 elasticsearch.username: elastic elasticsearch.password: changeme xpack.monitoring.ui.container.elasticsearch.enabled: true ``` **Sample of log** `2018-08-20 12:39:32.232321 Sql NoteLgAlw I Tec Transaction Started, Nested level: 1, MVCC Start Timestamp: 23249765 # # TID=5868 __DBMC_TransactionManager.h 318 Customized=0` ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/f/7f498d828ea533872670e76b4ec81875c32b0b15.png) **File encoding Unicode** ![2018-08-30_13-37-34](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0e477ff4cea52def6e149259ec89a28e81809816.png) **Same Log in Logstash** ![2018-08-30_13-44-48](https://us1.discourse-cdn.com/elastic/original/3X/8/6/86e1f3976e89fec1b7d80e73d8e447045ba37b46.png) ![2018-08-30_13-48-58](https://us1.discourse-cdn.com/elastic/original/3X/6/0/604d8ac85f52ec4006223b37187cc87518098193.png) **Same Log in Kibana** Here is how log is presented in kibana ![2018-08-30_13-51-03](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c2eefebd8d35c898c54863e37674550eec9886dc.png) ![2018-08-30_13-51-47](https://us1.discourse-cdn.com/elastic/original/3X/9/1/91a167c1f80e625ed2fa0c7674d4173aaac91d94.png) BUT, when i try filter by message, this will happend. ![2018-08-30_13-55-16](https://us1.discourse-cdn.com/elastic/original/3X/3/4/34f017c66eb5226563da084cb2a0cf4a4f9a6ab8.png) ![2018-08-30_13-55-42](https://us1.discourse-cdn.com/elastic/original/3X/2/3/234bf31502829695e440332e2e9bfadaaa31d2ad.png) ![2018-08-30_13-58-55](https://us1.discourse-cdn.com/elastic/original/3X/b/a/bad3bb2f714bdc6ef14f52ee3fa6ab64729abc65.png) NO MATCH AT ALL. I cant filter by massage at all if file from which is log harvested has set encoding as **unicode** **Here is behavior which i expect** ![2018-08-30_14-06-26](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c2eb4659c18dc7a3d75ceb24205f007491b93d89.png) **Same log same configs only thing that's different is encoding of file from which log is harvested** ![2018-08-30_14-05-25](https://us1.discourse-cdn.com/elastic/original/3X/b/a/babbddb95101e576e17690e38d759b8d16d75c28.png) But i cant use this as solution. The encoding of file must stay as Unicode. I tried to resolve this by set `encoding plain` . . . .`encoding utf-8` in filebeat.yml also tried set Logstash codec several variants `codec => plain { charset => "UTF-8" }` `codec => plain { charset => "UTF-16" }` `codec => plain { charset => "ASCII" }` . . . `codec => plain { charset => "ISO-8859-*" }` But no matter what i try or what i do result is still same... Can anyone please help me with this one ? Thank you! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 30, 2018, 1:46pm UTC](https://discuss.elastic.co/t/cant-filter-unicode-logs-at-all/146701/2 "2018-08-30T13:46:27Z")
Well, unfortunately "Unicode" isn't actually an encoding so it's not clear what kind of file you're actually getting from Windows. Judging by [https://stackoverflow.com/questions/13894898/unicode-file-in-notepad](https://stackoverflow.com/questions/13894898/unicode-file-in-notepad) I suggest you try utf-16le in your Filebeat configuration. ---
**Author:** ![yolt](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@yolt](https://discuss.elastic.co/u/yolt)\ **Post date:** [August 31, 2018, 9:32am UTC](https://discuss.elastic.co/t/cant-filter-unicode-logs-at-all/146701/3 "2018-08-31T09:32:17Z")
Thanks for your suggestion. When I set `encoding` to utf-16 **not** utf-16le in my filebeat.yml and remove `codec => plain { charset => "UTF-16" }` from logstash.conf it's **resolved** my issue. Funny, because i tried it before and it didn't work at all. but it was with 6.2.0 version of filebeat... never mind So thanks again . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 28, 2018, 9:32am UTC](https://discuss.elastic.co/t/cant-filter-unicode-logs-at-all/146701/4 "2018-09-28T09:32:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.