Dear Experts:
I have filebeat send logs (JSON messages) to the first Logstash server. From the first Logstash server I forward the logs to the second Logstash server using LumberJack.
The connectivity is working fine in that the logs are sent from the first to the second LS server successfully. However, in Kibana the logs (or the "message" in JSON format) cannot be parsed. It looks like the message is appended with the timestamp and the beat hostname at the beginning of the message. Example:
2018-07-03T15:56:08.489Z BeatHostName-01S {"Message":"System Started.","CreateDateUtc":"2018-07-03T09:40:22.8136608-06:00","TrackingId":"12345-56X9-491B-8D0B-9148FB8A0123","AppId":"09basdf-56bx-431b-8d0basdfasdf20151"}
Where 2018-07-03T15:56:08.489Z is the timestamp and BeatHostName-01S is the beat hostname.
My question is: how do I remove the timestamp and the beat hostname, or prevent it from being added to the original message?
Here are my Logstash config files on each Logstash server:
On the first Logstash server:
input {
beats {
port => 5044
}
}
output {
lumberjack {
hosts => ["secondLSserver"]
port => 1234
ssl_certificate => "c:/logstash.pub"
}
stdout { codec => rubydebug }
}
On the second Logstash server:
input {
lumberjack {
type => "MessageType"
port => 1234
ssl_certificate => "c:/logstash.pub"
ssl_key => "c:/logstash.key"
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["http://esserver:9200"]
manage_template => false
index => "MyIndex-%{+YYYY-MM}"
}
}