Log Encoding message incorrect

Elastic Stack Logstash
1

Hello,

Im new to elk. I'm using multiple inputs i) tcp ii) beats in my logstash.

i) For tcp , using the spring pattern with in the code itself and simply default value of logstash server IP
as below

And able to see the messgaes correctly without encoding issue.

ii) filebeats is where i want to forward logs to the same logstash from different instance.
i have installed the filebeats on the instance and edited the file beats.yml input(/var/log) & output as the logstash ip with 5:044.

My conf looks as below:

#input TCP for Microservices
#input Beats for the AEM instance server logs.

input {
tcp {
port => 5044
}
beats {
port => 5044
}
}

filter {

json {
source => "message"
}

}
output {
if [type] == "syslog" {
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
codec => "json"
}
}

else if [type][beats][hostname] == "ip-x-x-x-x" {
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
codec => "json"
document_type => "%{[@metadata][type]}"
}
}

}

When i see the messages on the kibana discover and search filter with IP address which is coming from beats servers messages looks broken as below.

message:\x81>L6\x9B\xF3\xF5\xBD\x83\xF1h2\xBB\x9C\xE9\xEC\xCA`\x916f\x87\xA5\xA5\xF5*\xDAݍ\x8EF\x93\xD1}Ãh<\x89\xE2,MN7"\x92\u0013>\rXFȌ\xBCE\f\xCF\u001C\x99"\xEFɛ\x87\x93!\x82Ee\u0013=|8\u0019^\e\u001DE\xF7\xEC\u001D\x8Do̢\xB

can someone please help me with this issue. Not sure if my conf with filter and codec doing right. But the logs comings from tcp input is looking good but not from beats

2

Hey @shri I'm moving this to the Logstash section, as it's more about ingesting your data properly with Logstash than Kibana.

3

Thanks Brandon.

4

input {
tcp {
port => 5044
}
beats {
port => 5044
}
}

Um, what? You can't have two input plugins listening on the same port.

5

ok i removed the beats from my input and only using the tcp to the port 5044. And i'm able to see logs from both. But, the instance logs seem to be still looking incorrrect.

message looking as below :
\x8F\u0014/#\x95\xFC\t3\x85\aL\x9D\xF8\u001F\t\xEB\u001C\xE6,"\xB1\xB0!\xD0#\u001E\xC5\xE1"\u001Cœ\xEFK\xB2\xEC) tags:_jsonparsefailure _id:yJXYkGEBw0V7GrzbeGcy _type:%{[@metadata][type]} _index:%{[@metadata][beat]}-2018.02.13

Also, this is what im seeing from logstash stdout

[2018-02-13T20:30:31,117][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\u000F3\xB2\u007F\xF4\xEC\xC1]\x81/\u0019\xC1\xC1D#t\xB0G\xCF\xE0\xE0\u0010'ܣg\xB8u0c\xAD~Dp\\xCAj\\xB3\\u0011f\\xE5\\xCC\\xD6Y\\xABVڈ\\xBB\\xF7\\x9F\\xDE}\\xBE\\u0013?N\\x9C\\xB7[;X\\xAD\\xD7Ki\\xD6\\xEB\\x8D\\xFE)\\xA8\\x8C\\xCAg\\u001F\\x8E\\xA8\\xEA\\u0014Ө|\\xE3\\xA3", :exception=>#<LogStash::Json::ParserError: Unexpected character ('\' (code 92)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: (byte[])"\u000F3\xB2\u007F\xF4\xEC\xC1]\x81/\u0019\xC1\xC1D#t\xB0G\xCF\xE0\xE0\u0010'Ü£g\xB8u0c\xAD~Dpî�¶\xCAj\xB3\u0011f\xE5\xCC\xD6Y\xABVÚ�\xBB\xF7\x9F\xDE}\xBE\u0013?N\x9C\xB7[;X\xAD\xD7Ki\xD6\xEB\x8D\xFE)\xA8\x8C\xCAg\u001F\x8E\xA8\xEA\u0014Ó¨|\xE3\xA3"; line: 1, column: 2]>}

6

If you want to collect logs with Filebeat you need a beats input, it's just that it can't use the same port as your tcp input.

7

How do i add another port for the tcp inputs?.

8

Re-add the beats input but choose another port number. The port number must be unused. Over and out.

9

Thank you. That did help.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.