FileBeat in Windows and LogStash in Linux

Elastic Stack Logstash
1

Hi,

I running FileBeat on Windows and LogStash on Linux.

When I'm sending a log file from FileBeat to LogStash ... LogStash is getting in utf8 and the grok filtering is not working. But when I'm sending the file from the linux system using the nc command, Logstash is getting in ASCII and everything is working well.

Does it a known issue?

Regards,

Shmuel
PS: if logs is needed I can provide.

2

I'm not sure what you mean here. Does your log file contain any bytes with the eight bit set, i.e. an ASCII value >127? If yes then it's not ASCII. If no then there's no difference between the ASCII representation of the text and the UTF-8 representation of the same text.

3

No ... where can I send I you the log ... they are little bit too big to put them here.

4

We don't need the whole log. A single line that exhibits the problem you describe will do. Make sure you format it as preformatted text when posting. The same line fed through hexdump -C could also be useful.

5

[2017-02-01T14:44:22,184][DEBUG][logstash.pipeline ] filter received {"event"=>{"@timestamp"=>2017-02-01T14:44:22.178Z, "@version"=>"1", "message"=>"\u0000D\u0000E\u0000B\u0000U\u0000G\u0000 \u00002\u00000\u00001\u00007\u0000-\u00000\u00001\u0000-\u00002\u00005\u0000 \u00002\u00002\u0000:\u00004\u00008\u0000:\u00004\u00000\u0000,\u00001\u00008\u00005\u0000 \u0000c\u0000l\u0000a\u0000s\u0000s\u0000:\u0000G\u0000a\u0000t\u0000e\u0000w\u0000a\u0000y\u0000s\u0000_\u0000J\u0000s\u0000o\u0000n\u0000P\u0000o\u0000s\u0000t\u0000G\u0000W\u0000 \u0000t\u0000o\u0000p\u0000i\u0000c\u0000:\u0000n\u0000u\u0000l\u0000l\u0000 \u0000m\u0000e\u0000t\u0000h\u0000o\u0000d\u0000:\u0000O\u0000n\u0000L\u0000o\u0000a\u0000d\u0000 \u0000s\u0000e\u0000r\u0000v\u0000e\u0000r\u0000:\u0000I\u0000P\u0000-\u00000\u0000A\u00000\u00000\u00000\u00001\u0000F\u00008\u0000 \u0000i\u0000p\u0000:\u00001\u00000\u0000.\u00000\u0000.\u00002\u0000.\u00005\u0000 \u0000r\u0000e\u0000q\u0000i\u0000d\u0000:\u0000f\u0000f\u00003\u00005\u0000f\u00008\u0000a\u00006\u0000-\u00007\u00001\u00003\u00005\u0000-\u00004\u00003\u00005\u00009\u0000-\u0000a\u0000c\u00009\u00007\u0000-\u00004\u0000e\u00009\u00007\u0000d\u0000a\u00000\u00004\u0000b\u00007\u0000a\u00003\u0000 \u0000p\u0000a\u0000r\u0000t\u0000n\u0000e\u0000r\u0000:\u00004\u00003\u00006\u0000 \u0000a\u0000c\u0000t\u0000i\u0000o\u0000n\u0000:\u0000G\u0000e\u0000t\u0000M\u0000e\u0000d\u0000i\u0000a\u0000I\u0000n\u0000f\u0000o\u0000 \u0000u\u0000i\u0000d\u0000:\u00000\u0000 \u0000m\u0000s\u0000g\u0000:\u0000A\u0000P\u0000I\u0000 \u0000R\u0000e\u0000q\u0000u\u0000e\u0000s\u0000t\u0000 \u0000-\u0000\r\u0000\n\u0000D\u0000E\u0000B\u0000U\u0000G\u0000 \u00002\u00000\u00001\u00007\u0000-\u00000\u00001\u0000-\u00002\u00005\u0000 \u00002\u00002\u0000:\u00004\u00008\u0000:\u00004\u00000\u0000,\u00001\u00008\u00005\u0000 \u0000c\u0000l\u0000a\u0000s\u0000s\u0000:\u0000G\u0000a\u0000t\u0000e\u0000w\u0000a\u0000y\u0000s\u0000_\u0000J\u0000s\u0000o\u0000n\u0000P\u0000o\u0000s\u0000t\u0000G\u0000W\u0000 \u0000t\u0000o\u0000p\u0000i\u0000c\u0000:\u0000n\u0000u\u0000l\u0000l\u0000 \u0000m\u0000e\u0000t\u0000h\u0000o\u0000d\u0000:\u0000O\u0000n\u0000L\u0000o\u0000a\u0000d\u0000 \u0000s\u0000e\u0000r\u0000v\u0000e\u0000r\u0000:\u0000I\u0000P\u0000-\u00000\u0000A\u00000\u00000\u00000\u00001\u0000F\u00008\u0000 \u0000i\u0000p\u0000:\u00001\u00000\u0000.\u00000\u0000.\u00002\u0000.\u00005\u0000 \u0000r\u0000e\u0000q\u0000i\u0000d\u0000:\u0000f\u0000f\u00003\u00005\u0000f\u00008\u0000a\u00006\u0000-\u00007\u00001\u00003\u00005\u0000-\u00004\u00003\u00005\u00009\u0000-\u0000a\u0000c\u00009\u00007\u0000-\u00004\u0000e\u00009\u00007\u0000d\u0000a\u00000\u00004\u0000b\u00007\u0000a\u00003\u0000 \u0000p\u0000a\u0000r\u0000t\u0000n\u0000e\u0000r\u0000:\u00004\u00003\u00006\u0000 \u0000a\u0000c\u0000t\u0000i\u0000o\u0000n\u0000:\u0000G\u0000e\u0000t\u0000M\u0000e\u0000d\u0000i\u0000a\u0000I\u0000n\u0000f\u0000o\u0000 \u0000u\u0000i\u0000d\u0000:\u00000\u0000 \u0000m\u0000s\u0000g\u0000:\u0000A\u0000P\u0000I\u0000 \u0000R\u0000e\u0000q\u0000u\u0000e\u0000s\u0000t\u0000 \u0000-\u0000\r\u0000", "tags"=>["multiline"]}}

6

[2017-02-01T14:44:22,185][DEBUG][logstash.filters.grok ] Running grok filter {:event=>2017-02-01T14:44:22.178Z %{host} ...
[2017-02-01T14:44:22,190][DEBUG][logstash.pipeline ] output received {"event"=>{"@timestamp"=>2017-02-01T14:44:22.178Z, "@version"=>"1", "message"=>"\u0000D\u0000E\u0000B\u0000U\u0000G\u0000 \u00002\u00000\u00001\u00007\u0000-\u00000\u00001\u0000-\u00002\u00005\u0000 \u00002\u00002\u0000:\u00004\u00008\u0000:\u00004\u00000\u0000,\u00001\u00008\u00005\u0000 \u0000c\u0000l\u0000a\u0000s\u0000s\u0000:\u0000G\u0000a\u0000t\u0000e\u0000w\u0000a\u0000y\u0000s\u0000_\u0000J\u0000s\u0000o\u0000n\u0000P\u0000o\u0000s\u0000t\u0000G\u0000W\u0000 \u0000t\u0000o\u0000p\u0000i\u0000c\u0000:\u0000n\u0000u\u0000l\u0000l\u0000 \u0000m\u0000e\u0000t\u0000h\u0000o\u0000d\u0000:\u0000O\u0000n\u0000L\u0000o\u0000a\u0000d\u0000 \u0000s\u0000e\u0000r\u0000v\u0000e\u0000r\u0000:\u0000I\u0000P\u0000-\u00000\u0000A\u00000\u00000\u00000\u00001\u0000F\u00008\u0000 \u0000i\u0000p\u0000:\u00001\u00000\u0000.\u00000\u0000.\u00002\u0000.\u00005\u0000 \u0000r\u0000e\u0000q\u0000i\u0000d\u0000:\u0000f\u0000f\u00003\u00005\u0000f\u00008\u0000a\u00006\u0000-\u00007\u00001\u00003\u00005\u0000-\u00004\u00003\u00005\u00009\u0000-\u0000a\u0000c\u00009\u00007\u0000-\u00004\u0000e\u00009\u00007\u0000d\u0000a\u00000\u00004\u0000b\u00007\u0000a\u00003\u0000 \u0000p\u0000a\u0000r\u0000t\u0000n\u0000e\u0000r\u0000:\u00004\u00003\u00006\u0000 \u0000a\u0000c\u0000t\u0000i\u0000o\u0000n\u0000:\u0000G\u0000e\u0000t\u0000M\u0000e\u0000d\u0000i\u0000a\u0000I\u0000n\u0000f\u0000o\u0000 \u0000u\u0000i\u0000d\u0000:\u00000\u0000 \u0000m\u0000s\u0000g\u0000:\u0000A\u0000P\u0000I\u0000 \u0000R\u0000e\u0000q\u0000u\u0000e\u0000s\u0000t\u0000 \u0000-\u0000\r\u0000\n\u0000D\u0000E\u0000B\u0000U\u0000G\u0000 \u00002\u00000\u00001\u00007\u0000-\u00000\u00001\u0000-\u00002\u00005\u0000 \u00002\u00002\u0000:\u00004\u00008\u0000:\u00004\u00000\u0000,\u00001\u00008\u00005\u0000 \u0000c\u0000l\u0000a\u0000s\u0000s\u0000:\u0000G\u0000a\u0000t\u0000e\u0000w\u0000a\u0000y\u0000s\u0000_\u0000J\u0000s\u0000o\u0000n\u0000P\u0000o\u0000s\u0000t\u0000G\u0000W\u0000 \u0000t\u0000o\u0000p\u0000i\u0000c\u0000:\u0000n\u0000u\u0000l\u0000l\u0000 \u0000m\u0000e\u0000t\u0000h\u0000o\u0000d\u0000:\u0000O\u0000n\u0000L\u0000o\u0000a\u0000d\u0000 \u0000s\u0000e\u0000r\u0000v\u0000e\u0000r\u0000:\u0000I\u0000P\u0000-\u00000\u0000A\u00000\u00000\u00000\u00001\u0000F\u00008\u0000 \u0000i\u0000p\u0000:\u00001\u00000\u0000.\u00000\u0000.\u00002\u0000.\u00005\u0000 \u0000r\u0000e\u0000q\u0000i\u0000d\u0000:\u0000f\u0000f\u00003\u00005\u0000f\u00008\u0000a\u00006\u0000-\u00007\u00001\u00003\u00005\u0000-\u00004\u00003\u00005\u00009\u0000-\u0000a\u0000c\u00009\u00007\u0000-\u00004\u0000e\u00009\u00007\u0000d\u0000a\u00000\u00004\u0000b\u00007\u0000a\u00003\u0000 \u0000p\u0000a\u0000r\u0000t\u0000n\u0000e\u0000r\u0000:\u00004\u00003\u00006\u0000 \u0000a\u0000c\u0000t\u0000i\u0000o\u0000n\u0000:\u0000G\u0000e\u0000t\u0000M\u0000e\u0000d\u0000i\u0000a\u0000I\u0000n\u0000f\u0000o\u0000 \u0000u\u0000i\u0000d\u0000:\u00000\u0000 \u0000m\u0000s\u0000g\u0000:\u0000A\u0000P\u0000I\u0000 \u0000R\u0000e\u0000q\u0000u\u0000e\u0000s\u0000t\u0000 \u0000-\u0000\r\u0000", "tags"=>["multiline", "_grokparsefailure"]}}

7

And the line sent is:
DEBUG 2017-01-25 22:48:40,185 class:Gateways_JsonPostGW topic:null method:OnLoad server:IP-0A0001F8 ip:10.0.2.5 reqid:ff35f8a6-7135-4359-ac97-4e97da04b7a3 partner:436 action:GetMediaInfo uid:0 msg:API Request -

8

Isn't this a UTF-16 file? It looks like you should set Filebeat's encoding option to utf-16be.

9

I have set the encoding in the input but not in the output:
filebeat.prospectors:

  • input_type: log
    paths:
    • D:\Logs\v4_1\tvp_api*.log
      encoding: plain
      ignore_older: 24h
      document_type: ws_log
      multiline.pattern: '^DEBUG'
      multiline.negate: true
      multiline.match: after
      tail_files: false
      hosts: ["localhost:9200"]
      output.logstash:
      enabled: true
      hosts: ["35.157.114.116:7002"]
      loadbalance: true

output.file:
enabled: false
path: D:\Logs\filebeat
filename: filebeat_out
number_of_files: 7
logging.level: debug
logging.to_syslog: false
logging.to_files: true
logging.files:
path: D:\Logs\filebeat
name: filebeat
rotateeverybytes: 10485760 # = 10MB
keepfiles: 10

10

Setting encoding to plain won't do any good. As I said you should probably use utf-16be.

11

I have set it in filebeat and in the logstash see below the configuration file:
input {
tcp {
port => 7002
codec => multiline {
pattern => "^(DEBUG|INFO|ERROR|TRACE|WARN|FATAL)"
negate => "true"
what => "previous"
charset => "UTF-16BE"
}
}
}
filter {
grok {
match => {
message => "^%{LOGLEVEL:logLevel} %{TIMESTAMP_ISO8601:sourceTimestamp} class:%{DATA:class} topic:%{DATA:topic} method:%{DATA:method} server:%{DATA:server} ip:%{IPORHOST:ip} reqid:%{DATA:reqid} partner:%{DATA:partner} action:%{DATA:action} uid:%{NUMBER:userId} msg:%{GREEDYDATA:msg}"
}
remove_field => [ "count", "fields", "offset", "message" ]
}

date {

match => [ "sourceTimestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]

target => "@timestamp"

remove_field => [ "sourceTimestamp", "message" ]

}

}
output {
elasticsearch {
hosts => ["elastic:9200"]
index => "ott-logs-%{+YYYY.MM.dd}"
}
}

12

I have got Chinese character and it's still not working:
[2017-02-01T15:37:14,622][DEBUG][logstash.pipeline ] filter received {"event"=>{"@timestamp"=>2017-02-01T15:37:14.613Z, "port"=>64109, "@version"=>"1", "host"=>"52.19.166.128", "message"=>"㉗\u0000\u0001㉃\u0000ŷ硞泐八�【߰衻뚌摛煲俭�ᙒ㨖웆怘쳅㺻˛刭㦥葼蟑瑻嫟\u0E8E믿뭟굖㞫헍�踜ꦥ䢀枈⾞ǡ㧔菫ↁ⍓҄츎籤詰䧠㛚酃ꓑ͂ꙴ⥕⚕ﺮ෦ᦚ䶪췺ᜤნ㈷꿋㹢唽렾哕ꦨ畕얓꿉�次쥛嚶籜充뢆몮ଜdz㰯쫤㾓鷼ያ뿒랉酃ꃾ\u1AF6辝硣榙ᦑ敘沰傉�ᣑఔˮ⣲㎽蓺㺸䋜ﴔ텹�ಃᠹ㺺ᚿ丏蹚ᅸ㻱賻荔睊⧽礣걇굒镦ꤑ㌿�ᚻ⸷�蛖닔릑䕮뚒驭⤋�阭ꧢ塒⸼쵱譼ⶨ觖\u4DB8蕛䯻ꥳ抱\u2D2Aㆆữ\u0E7B鼖๑䡈稼썄跖쭿ᙈ藸滣쑳끮Ȅ鏪㒃쯥伀\u0000\uFFFF席飊"}}
[2017-02-01T15:37:14,624][DEBUG][logstash.filters.grok ] Running grok filter {:event=>2017-02-01T15:37:14.613Z 52.19.166.128 ㉗㉃ŷ硞泐八�【߰衻뚌摛煲俭�ᙒ㨖웆怘쳅㺻˛刭㦥葼蟑瑻嫟ຎ믿뭟굖㞫헍�踜ꦥ䢀枈⾞ǡ㧔菫ↁ⍓҄츎籤詰䧠㛚酃ꓑ͂ꙴ⥕⚕ﺮ෦ᦚ䶪췺ᜤნ㈷꿋㹢唽렾哕ꦨ畕얓꿉�次쥛嚶籜充뢆몮ଜdz㰯쫤㾓鷼ያ뿒랉酃ꃾ᫶辝硣榙ᦑ敘沰傉�ᣑఔˮ⣲㎽蓺㺸䋜ﴔ텹�ಃᠹ㺺ᚿ丏蹚ᅸ㻱賻荔睊⧽礣걇굒镦ꤑ㌿�ᚻ⸷�蛖닔릑䕮뚒驭⤋�阭ꧢ塒⸼쵱譼ⶨ觖䶸蕛䯻ꥳ抱⴪ㆆữ๻鼖๑䡈稼썄跖쭿ᙈ藸滣쑳끮Ȅ鏪㒃쯥伀▒席飊}
[2017-02-01T15:37:14,628][DEBUG][logstash.filters.grok ] Event now: {:event=>2017-02-01T15:37:14.613Z 52.19.166.128 ㉗㉃ŷ硞泐八�【߰衻뚌摛煲俭�ᙒ㨖웆怘쳅㺻˛刭㦥葼蟑瑻嫟ຎ믿뭟굖㞫헍�踜ꦥ䢀枈⾞ǡ㧔菫ↁ⍓҄츎籤詰䧠㛚酃ꓑ͂ꙴ⥕⚕ﺮ෦ᦚ䶪췺ᜤნ㈷꿋㹢唽렾哕ꦨ畕얓꿉�次쥛嚶籜充뢆몮ଜdz㰯쫤㾓鷼ያ뿒랉酃ꃾ᫶辝硣榙ᦑ敘沰傉�ᣑఔˮ⣲㎽蓺㺸䋜ﴔ텹�ಃᠹ㺺ᚿ丏蹚ᅸ㻱賻荔睊⧽礣걇굒镦ꤑ㌿�ᚻ⸷�蛖닔릑䕮뚒驭⤋�阭ꧢ塒⸼쵱譼ⶨ觖䶸蕛䯻ꥳ抱⴪ㆆữ๻鼖๑䡈稼썄跖쭿ᙈ藸滣쑳끮Ȅ鏪㒃쯥伀▒席飊}
[2017-02-01T15:37:14,634][DEBUG][logstash.pipeline ] output received {"event"=>{"@timestamp"=>2017-02-01T15:37:14.613Z, "port"=>64109, "@version"=>"1", "host"=>"52.19.166.128", "message"=>"㉗\u0000\u0001㉃\u0000ŷ硞泐八�【߰衻뚌摛煲俭�ᙒ㨖웆怘쳅㺻˛刭㦥葼蟑瑻嫟\u0E8E믿뭟굖㞫헍�踜ꦥ䢀枈⾞ǡ㧔菫ↁ⍓҄츎籤詰䧠㛚酃ꓑ͂ꙴ⥕⚕ﺮ෦ᦚ䶪췺ᜤნ㈷꿋㹢唽렾哕ꦨ畕얓꿉�次쥛嚶籜充뢆몮ଜdz㰯쫤㾓鷼ያ뿒랉酃ꃾ\u1AF6辝硣榙ᦑ敘沰傉�ᣑఔˮ⣲㎽蓺㺸䋜ﴔ텹�ಃᠹ㺺ᚿ丏蹚ᅸ㻱賻荔睊⧽礣걇굒镦ꤑ㌿�ᚻ⸷�蛖닔릑䕮뚒驭⤋�阭ꧢ塒⸼쵱譼ⶨ觖\u4DB8蕛䯻ꥳ抱\u2D2Aㆆữ\u0E7B鼖๑䡈稼썄跖쭿ᙈ藸滣쑳끮Ȅ鏪㒃쯥伀\u0000\uFFFF席飊", "tags"=>["_grokparsefailure"]}}

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.