Logstash output syslog : how to remove added {host} field?

[Travis]

Hello there !

I have to forward kafka logs to syslog relay. I know it's weird but I have to.

Here is logstash conf :

#kafka input
input {
  kafka {
    topics => ["test"]
    codec => json
    bootstrap_servers => "kafka1:9092"

#keep only message of log
filter
{
prune {
  whitelist_names => ["^message$"]
  }
}


#output to syslog 
output {
  syslog {
    host => "1.2.3.4"
    port => 514
    protocol => "tcp"
    appname => ""
    msgid => ""
    sourcehost => ""
    procid => ""
  }
}

But this was not enough, so I edited syslog output plugin logstash-output-syslog-3.0.5 (syslog.rb) :

Full plugin code is here : https://github.com/logstash-plugins/logstash-output-syslog/blob/master/lib/logstash/outputs/syslog.rb

[...]

  def publish(event, payload)
    appname = event.sprintf(@appname)
    procid = event.sprintf(@procid)
    sourcehost = event.sprintf(@sourcehost)

    message = payload.to_s.rstrip.gsub(/[\r][\n]/, "\n").gsub(/[\n]/, '\n')

    # fallback to pri 13 (facility 1, severity 5)
    if @use_labels
      facility_code = (FACILITY_LABELS.index(event.sprintf(@facility)) || 1)
      severity_code = (SEVERITY_LABELS.index(event.sprintf(@severity)) || 5)
      priority = (facility_code * 8) + severity_code
    else
      priority = Integer(event.sprintf(@priority)) rescue 13
      priority = 13 if (priority < 0 || priority > 191)
    end

    if @is_rfc3164
      timestamp = event.sprintf("%{+MMM dd HH:mm:ss}")
#original code	  
#      syslog_msg = "<#{priority.to_s}>#{timestamp} #{sourcehost} #{appname}[#{procid}]: #{message}"

#new code to have only priority and message 
      syslog_msg = "<#{priority.to_s}>#{message}"
    else
      msgid = event.sprintf(@msgid)
      timestamp = event.sprintf("%{+YYYY-MM-dd'T'HH:mm:ss.SSSZZ}")
      syslog_msg = "<#{priority.to_s}>1 #{timestamp} #{sourcehost} #{appname} #{procid} #{msgid} - #{message}"
      syslog_msg = "<#{priority.to_s}>#{message}"
    end

[...] 

When event is received on the syslog relay, I have this :

<13>%{host} Jan 5 16:42:29 server1 misc-centreon(misc_centreon)[61665]: INFO: running

But I expect this :

<13>Jan 5 16:42:29 server1 misc-centreon(misc_centreon)[61665]: INFO: running

Where do comes from {host} at the beginning of the message ? I can't succeed to find it in plugin code and remove it

Can you help me ?

Thanks !

[Badger]

You do not appear to be using any of the features of the syslog output. Why not replace it with a tcp output instead of rewriting the syslog output to do no enrichment of the message?

[Travis]

Indeed I don't use main syslog features but I have to respect a minimum the form as log will be processed like a log coming directly from a machine once in the syslog relay.

I just have to remove %{host} and I don't understand how to do it when looking at the code

[Travis]

As reported here -> Logstash syslog output ignores message it looks like there is like a bug and the workaround is to add the field "host" in filter to be taken into account by the plugin

So, for my needs I use this :

#keep only message of log and add empty host field
filter
{
prune {
  whitelist_names => ["^message$"]
  }
mutate {
  add_field => {"host" => ""}
  }
}

This way, received log by syslog relay is :

<13> Jan 5 16:42:29 server1 misc-centreon(misc_centreon)[61665]: INFO: running

Unfortunately, as you can see, there i still a space after <13> but it's already that !

I don't understand why it is so hard to find where does comes from this %{host} field. If someone has a better comprehension...

[mtudisco]

Hi,
The way i manged to solve my problem was using plain codec:

syslog {
		        host => "X.X.X.X"
		        port => 514
		        protocol => "tcp"
		        rfc => rfc5424
                sourcehost => "%{source_ip}"
                appname => "%{event_type}"
                codec => plain { format => "%{message}" }
	           }

hope it helps

[Travis]

Yes perfect... It works

Thanks a lot !