Can't filter Unicode logs at all


I need help to find out how to set Filebeat or Logstash or both... to be able successfully
filter logs by message from files which have set encoding as Unicode
I will provide screenshots to describe problem as best as i can.

Currently i'm using 6.4.0 version for Filebeat, Logstash, Elasticsearch and Kibana.

Logstash, Elasticsearch and Kibana are runing as docker services on one NODE.

  • Docker version 17.12.1-ce
  • OS linux Ubuntu 16.04 LTS

Filebeat is harvesting logs on Windows 10 as process and sending them to Logstash.

Here is my config for each utility.




    - type: log
          - C:\Users\C5260750\Desktop\customlogs\*

        level: debug
        status of machine: running
        review: 1

      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'

      multiline.negate: true

      multiline.match: after

      hosts: [""]



input {
beats {
port => 5044
output {
 elasticsearch {
  hosts => ""
  user => elastic
  password => changeme
stdout {
codec => rubydebug

logstash.yml ""
path.config: /usr/share/logstash/pipeline
xpack.monitoring.elasticsearch.url: [](
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme




kibana.yml kibana "0"
elasticsearch.username: elastic
elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: true

Sample of log

2018-08-20 12:39:32.232321 Sql NoteLgAlw I Tec Transaction Started, Nested level: 1, MVCC Start Timestamp: 23249765 # # TID=5868 __DBMC_TransactionManager.h 318 Customized=0

File encoding Unicode

Same Log in Logstash

Same Log in Kibana

Here is how log is presented in kibana

BUT, when i try filter by message, this will happend.



I cant filter by massage at all if file from which is log harvested has set encoding as unicode

Here is behavior which i expect

Same log same configs only thing that's different is encoding of file from which log is harvested

But i cant use this as solution. The encoding of file must stay as Unicode.

I tried to resolve this by set

encoding plain
.encoding utf-8

in filebeat.yml

also tried set Logstash codec several variants

codec => plain { charset => "UTF-8" }
codec => plain { charset => "UTF-16" }
codec => plain { charset => "ASCII" }
codec => plain { charset => "ISO-8859-*" }

But no matter what i try or what i do result is still same...

Can anyone please help me with this one ?

Thank you!

Well, unfortunately "Unicode" isn't actually an encoding so it's not clear what kind of file you're actually getting from Windows. Judging by I suggest you try utf-16le in your Filebeat configuration.

1 Like

Thanks for your suggestion.
When I set encoding to utf-16 not utf-16le in my filebeat.yml and remove codec => plain { charset => "UTF-16" } from logstash.conf it's resolved my issue. Funny, because i tried it before and it didn't work at all. but it was with 6.2.0 version of filebeat... never mind

So thanks again .

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.