Filebeat - received an event - has different character encoding

j123 · January 29, 2016, 6:43pm

Hello,

I am using filebeat (Linux box) to send logs to ELK - logstash., It is not sending UTF-8 (Linux) encoding. Looks like hexa-decimal. Help please.

error message in logstash.log:

{:timestamp=>"2016-01-29T11:47:40.722000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\u0014\u0006X\u0010E\u001D@B\xCA\u0010\x9D\xED{_=z\xF8r2\x99\x9C\xD1\xD7\xEDaaU\xC4\u0016#B\xBD\x87\x8D\xC2\b5tڨ\xC3o", :expected_charset=>"UTF-8", :level=>:warn}

Message in elasticsearch:

message:]\u0013\xA7X\xF8#\xAA\u001D>T\xF1߹\u0014\u0006X\u0010E\u001D@B\xCA\u0010\x9D\xED{_=z\xF8r2\x99\x9C\xD1\xD7\xEDaaU\xC4\u0016#B\xBD\x87\x8D\xC2\b5tڨ\xC3o

--
config

filebeat ver: filebeat-1.0.1-1.x86_64 (on linux)

logstash.conf - configured for plain/UTF-8.
(setting the codec/charset -makes no difference)

input {
tcp {

        type => "dev-mail4"
         port => 3570
            codec => plain {
                charset => "UTF-8"
        }
}

}

ruflin · February 1, 2016, 8:15am

What is the encoding of the file you are reading?

steffens · February 1, 2016, 10:45am

filebeat -> logstash uses json and therefore requires to encode content into UTF-8 on sender side. This hex-like dump results from raw-content being read which is clearly not UTF-8. What's the encoding of the file you are trying to send?

j123 · February 8, 2016, 3:38pm

Linux syslog, and maillogs..

filebeat.yml:
encoding: plain (tried UTF-8 same result)
document_type: log

steffens · February 9, 2016, 12:54am

plain and utf-8 are basically the same when forwarding to logstash. Difference is the utf-8 encoding applies the hex-encodings of unknown characters at read time.

steffens · February 9, 2016, 12:57am

Can you share some content of original log-files with us for testing? Some original file is required. Just copy'n pasting in forum will kinda 'fix' the encoding.

athreya_vc · February 25, 2016, 3:09pm

Hi,

I am facing the same issue with the filebeats+logstash, for postfix maillog I am facing the same error.

On the filebeat, I setup encoding to UTF-8. On the logstash side,

input {
  beats {
    port => 5044
    type => "maillog"
    codec => {
                 charset => "UTF-8" }
  }
  tcp {
        port => 5544
      }
}

Regards,
A

steffens · February 26, 2016, 7:09am

Use the encoding in filebeat, not in logstash.
both, filebeat and logstash assume input to be UTF-8 and setting charset to UTF-8 is mostly a simple copy of your input data (masking invalid characters). The encoding must match the file encoding you read from.
looks like some kind of misconfiguration, but without getting some original logs (raw file) reproducing the error I have a hard time seeing what the fault is.

This happens for every single log line? If so, can you send a test mail and share the single log line in a separate file for testing?

athreya_vc · February 26, 2016, 12:19pm

Hi,

I did enable the encoding on the filebeat

filebeat:

List of prospectors to fetch data.

prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
# Paths that should be crawled and fetched. Glob based paths.
# To fetch all ".log" files from a specific level of subdirectories
# /var/log//.log can be used.
# For each file found under this path, a harvester is started.
# Make sure not file is defined twice as this can lead to unexpected behaviour.
paths:
- /var/log/maillog
#- c:\programdata\elasticsearch\logs*

  # Configure the file encoding for reading files with international characters
  # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
  # Some sample encodings:
  #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
  #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
  encoding: utf-8

I get some output in debug mode, but the message is "message" => "3\xB2\xE1m\u000F\xE0\x92J,"

like this

Regards,

A

athreya_vc · February 26, 2016, 1:34pm

This is my filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - /var/log/maillog
      input_type: log
      encoding: "utf-8"
      document_type: my_log
output:
  logstash:
    hosts: ["192.168.1.149:5544"]
  console:
    pretty: true
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

This is my logstash.conf

input {
        beats {
        port => 5044
        codec => plain { charset => "UTF-8" }
        type => "maillog"
     }
    tcp {
         port => 5544
      }
}
filter {
  if [type] == "maillog" {
    grok {
      match => { "message" => ["%{PF}", "%{DOVECOT}" ] }
    }
    date {
      match => [ "timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss" ]
    }
  }
  # I wanted to monitor metrics and health of logstash
  metrics {
    meter => "events"
    add_tag => "metric"
  }
}

output {
        stdout { codec => rubydebug}
    }

Regards,

A

steffens · February 26, 2016, 1:37pm

Encoding must match the content in the log file. Is the file/line really utf-8?

I really need a sample for reproducing this problem to help. PM me, if you want to arrange for sending me some data in private (+ gpg-encrypted).

athreya_vc · February 26, 2016, 2:24pm

Sure, let me check that.

file -bi /var/log/maillog
text/plain; charset=us-ascii

This is what I see.

Regards,

A

steffens · February 26, 2016, 5:49pm

us-ascii should be fully covered by utf-8. Are all lines garbled?

athreya_vc · February 29, 2016, 9:29am

Yes All the lines in the "Message" are Garbled.

steffens · February 29, 2016, 3:40pm

I can not reproduce the problem by copying the content from given link.

Let's try having filebeat printing to console only for testing first:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/maillog
      input_type: log
      document_type: my_log
output:
  console:
    pretty: true
shipper:

run filebeat in console with:

$ filebeat -e -c test.yml

athreya_vc · March 10, 2016, 4:42pm

filebeat -e -c test.yml

Works fine

athreya_vc · March 10, 2016, 4:59pm

  "@timestamp": "2016-03-10T16:51:50.404Z",
  "beat": {
    "hostname": mailserver.testlab.com,
    "name": mailserver.testlab.com
  },
  "count": 1,
  "fields": null,
  "input_type": "log",
  "message": "Mar 10 09:23:03 mailserver postfix/qmgr[28149]: 625EEC05FE: from=\u003cno-reply@testlab.com\u003e, size=1439, nrcpt=1 (queue active)",
  "offset": 3605648,
  "source": "/var/log/maillog",
  "type": "my_log"
}

ruflin · March 11, 2016, 11:02am

If the output to console works it looks more like a problem on the logstash side. Do you have any chance to try sending the log files directly to elasticsearch and check if they are there in the right format?

athreya_vc · March 21, 2016, 8:47am

I will try and update.

Regards,

A

steffens · March 21, 2016, 11:19am

with console being fine, let's try most minimal filebeat -> logstash:

test2.yml:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/maillog
      input_type: log
      document_type: my_log
output:
  logstash:
    hosts: ["localhost:5044"]

logstash.test.conf

input {
  beats {
    port => 5044
  }
}

output {
        stdout { codec => rubydebug}
}

run logstash with:

$ logstash agent -f logstash.test.conf

and filebeat with:

$ filebeat -e -c test2.yml

Topic		Replies	Views
FileBeat in Windows and LogStash in Linux Logstash	12	2589	March 1, 2017
LogStash encoding Issue from Filebeat IIS Access Logs 7.4.0 Stack Logstash	6	846	November 13, 2019
Character encoding problems Filebeat & Logstash Logstash	3	15797	November 8, 2018
Picking the right charset for Filebeat Logstash	1	849	February 11, 2018
Filebeat is not pushing nginx logs to logstash in correct format Beats filebeat	5	203	May 22, 2024

Filebeat - received an event - has different character encoding

-- config

List of prospectors to fetch data.

Related topics

--
config