Hello there !
I have to forward kafka logs to syslog relay. I know it's weird but I have to.
Here is logstash conf :
#kafka input
input {
kafka {
topics => ["test"]
codec => json
bootstrap_servers => "kafka1:9092"
#keep only message of log
filter
{
prune {
whitelist_names => ["^message$"]
}
}
#output to syslog
output {
syslog {
host => "1.2.3.4"
port => 514
protocol => "tcp"
appname => ""
msgid => ""
sourcehost => ""
procid => ""
}
}
But this was not enough, so I edited syslog output plugin logstash-output-syslog-3.0.5 (syslog.rb) :
Full plugin code is here : https://github.com/logstash-plugins/logstash-output-syslog/blob/master/lib/logstash/outputs/syslog.rb
[...]
def publish(event, payload)
appname = event.sprintf(@appname)
procid = event.sprintf(@procid)
sourcehost = event.sprintf(@sourcehost)
message = payload.to_s.rstrip.gsub(/[\r][\n]/, "\n").gsub(/[\n]/, '\n')
# fallback to pri 13 (facility 1, severity 5)
if @use_labels
facility_code = (FACILITY_LABELS.index(event.sprintf(@facility)) || 1)
severity_code = (SEVERITY_LABELS.index(event.sprintf(@severity)) || 5)
priority = (facility_code * 8) + severity_code
else
priority = Integer(event.sprintf(@priority)) rescue 13
priority = 13 if (priority < 0 || priority > 191)
end
if @is_rfc3164
timestamp = event.sprintf("%{+MMM dd HH:mm:ss}")
#original code
# syslog_msg = "<#{priority.to_s}>#{timestamp} #{sourcehost} #{appname}[#{procid}]: #{message}"
#new code to have only priority and message
syslog_msg = "<#{priority.to_s}>#{message}"
else
msgid = event.sprintf(@msgid)
timestamp = event.sprintf("%{+YYYY-MM-dd'T'HH:mm:ss.SSSZZ}")
syslog_msg = "<#{priority.to_s}>1 #{timestamp} #{sourcehost} #{appname} #{procid} #{msgid} - #{message}"
syslog_msg = "<#{priority.to_s}>#{message}"
end
[...]
When event is received on the syslog relay, I have this :
<13>%{host} Jan 5 16:42:29 server1 misc-centreon(misc_centreon)[61665]: INFO: running
But I expect this :
<13>Jan 5 16:42:29 server1 misc-centreon(misc_centreon)[61665]: INFO: running
Where do comes from {host} at the beginning of the message ? I can't succeed to find it in plugin code and remove it
Can you help me ?
Thanks !