Centos 7.5 with rsyslog 8.24 / logstash 6.3.0 - on logstash output always severity code 5 and facility code 1 - Other post found on but still not solved


(Peter Garlic) #1

Hello, I´m a ELK beginner and from one week I'm working on strange problem.

I installed one ELK test infrastructure based on Centos 7 (7.5.1804) and we have to send data to logstash (logstash-6.3.0-1) usysing rsyslog (rsyslog-8.24.0-16.el7_5.4.x86_64).

After installation Kibana showed a data problem and after a deep analisys I found the problem on Logstash: When I parse rsyslog with logstash the output hav always syslog_severity_code: 5 (syslog_severity: notice) and syslog_facility_code: 1 (syslog_facility: user-level).

Test session made with rsyslog as receiver and tcpdump show me that the incoming data is correct but logstash cannot decode the data correctly.

** This is the rsyslog sender configuration: **

$SystemLogSocketName /run/systemd/journal/syslog
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.
/var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
.emerg :omusrmsg:
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
. action(type="omfwd" target="192.168.56.103" port="1514" protocol="tcp")

This is the logstash chain:

input {
  tcp { # Linux Centos 7 syslog
    type => "syslog"
    port => 1514
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  if [type] == "syslog" {
    file { 
      path => "/var/log/%{+YYYY-MM-dd}-syslog.log" 
      codec => line { format => "%{message}"} 
    }
  } else {
    file {
      path => "/var/log/%{+YYYY-MM-dd}-other.log" 
      codec => line { format => "custom: %{message}"}
    }
  }
   stdout { codec => rubydebug { metadata => true } }
}

output with rubydebug
If I enable metadata on output we got one output that show as the log is received correctly, but not decoded on the right format:

{
    "syslog_severity_code" => 5,
               "@metadata" => {
        "ip_address" => "192.168.56.104"
    },
         "syslog_facility" => "user-level",
    "syslog_facility_code" => 1,
          "syslog_program" => "systemd",
                 "message" => "<31>Jun 22 08:56:10 server75-104 systemd: systemd-journald.service: added fd to fd store.",
                    "type" => "syslog",
          "syslog_message" => "systemd-journald.service: added fd to fd store.",
         "syslog_severity" => "notice",
           "received_from" => "centos7504",
              "@timestamp" => 2018-06-22T06:56:10.000Z,
         "syslog_hostname" => "server75-104",
                    "port" => 58498,
             "received_at" => "2018-06-22T06:56:59.191Z",
        "syslog_timestamp" => "Jun 22 08:56:10",
                    "host" => "centos7504",
                "@version" => "1"
}
{
    "syslog_severity_code" => 5,
               "@metadata" => {
        "ip_address" => "192.168.56.104"
    },
         "syslog_facility" => "user-level",
    "syslog_facility_code" => 1,
          "syslog_program" => "kernel",
                 "message" => "<6>Jun 22 08:56:10 server75-104 kernel: device-mapper: uevent: version 1.0.3",
                    "type" => "syslog",
          "syslog_message" => "device-mapper: uevent: version 1.0.3",
         "syslog_severity" => "notice",
           "received_from" => "centos7504",
              "@timestamp" => 2018-06-22T06:56:10.000Z,
         "syslog_hostname" => "server75-104",
                    "port" => 58498,
             "received_at" => "2018-06-22T06:56:59.191Z",
        "syslog_timestamp" => "Jun 22 08:56:10",
                    "host" => "centos7504",
                "@version" => "1"
}
{
    "syslog_severity_code" => 5,
               "@metadata" => {
        "ip_address" => "192.168.56.104"
    },
         "syslog_facility" => "user-level",
    "syslog_facility_code" => 1,
          "syslog_program" => "kernel",
                 "message" => "<6>Jun 22 08:56:10 server75-104 kernel: device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com",
                    "type" => "syslog",
          "syslog_message" => "device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com",
         "syslog_severity" => "notice",
           "received_from" => "centos7504",
              "@timestamp" => 2018-06-22T06:56:10.000Z,
         "syslog_hostname" => "server75-104",
                    "port" => 58498,
             "received_at" => "2018-06-22T06:56:59.191Z",
        "syslog_timestamp" => "Jun 22 08:56:10",
                    "host" => "centos7504",
                "@version" => "1"

}

I checked out other similar post on this forum, but until now I didn't find a solution. :roll_eyes:

Someone can help me please? :grimacing:
-Thanks


Centos 7 / rsyslog / logstash 6.3.0 - on logstash output always severity code 5 and facility code 1 (2)
Centos 7 / rsyslog / logstash 6.3.0 - on logstash output always severity code 5 and facility code 1 (2)
(Peter Garlic) #3

....continued in a new post....

Centos 7 / rsyslog / logstash 6.3.0 - on logstash output always severity code 5 and facility code 1 (2)


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.