Hello, I´m a ELK beginner and from one week I'm working on strange problem.
I installed one ELK test infrastructure based on Centos 7 (7.5.1804) and we have to send data to logstash (logstash-6.3.0-1) usysing rsyslog (rsyslog-8.24.0-16.el7_5.4.x86_64).
After installation Kibana showed a data problem and after a deep analisys I found the problem on Logstash: When I parse rsyslog with logstash the output hav always syslog_severity_code: 5 (syslog_severity: notice) and syslog_facility_code: 1 (syslog_facility: user-level).
Test session made with rsyslog as receiver and tcpdump show me that the incoming data is correct but logstash cannot decode the data correctly.
** This is the rsyslog sender configuration: **
$SystemLogSocketName /run/systemd/journal/syslog
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv. /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
.emerg :omusrmsg:
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
. action(type="omfwd" target="192.168.56.103" port="1514" protocol="tcp")
This is the logstash chain:
input {
tcp { # Linux Centos 7 syslog
type => "syslog"
port => 1514
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
if [type] == "syslog" {
file {
path => "/var/log/%{+YYYY-MM-dd}-syslog.log"
codec => line { format => "%{message}"}
}
} else {
file {
path => "/var/log/%{+YYYY-MM-dd}-other.log"
codec => line { format => "custom: %{message}"}
}
}
stdout { codec => rubydebug { metadata => true } }
}
output with rubydebug
If I enable metadata on output we got one output that show as the log is received correctly, but not decoded on the right format:
{
"syslog_severity_code" => 5,
"@metadata" => {
"ip_address" => "192.168.56.104"
},
"syslog_facility" => "user-level",
"syslog_facility_code" => 1,
"syslog_program" => "systemd",
"message" => "<31>Jun 22 08:56:10 server75-104 systemd: systemd-journald.service: added fd to fd store.",
"type" => "syslog",
"syslog_message" => "systemd-journald.service: added fd to fd store.",
"syslog_severity" => "notice",
"received_from" => "centos7504",
"@timestamp" => 2018-06-22T06:56:10.000Z,
"syslog_hostname" => "server75-104",
"port" => 58498,
"received_at" => "2018-06-22T06:56:59.191Z",
"syslog_timestamp" => "Jun 22 08:56:10",
"host" => "centos7504",
"@version" => "1"
}
{
"syslog_severity_code" => 5,
"@metadata" => {
"ip_address" => "192.168.56.104"
},
"syslog_facility" => "user-level",
"syslog_facility_code" => 1,
"syslog_program" => "kernel",
"message" => "<6>Jun 22 08:56:10 server75-104 kernel: device-mapper: uevent: version 1.0.3",
"type" => "syslog",
"syslog_message" => "device-mapper: uevent: version 1.0.3",
"syslog_severity" => "notice",
"received_from" => "centos7504",
"@timestamp" => 2018-06-22T06:56:10.000Z,
"syslog_hostname" => "server75-104",
"port" => 58498,
"received_at" => "2018-06-22T06:56:59.191Z",
"syslog_timestamp" => "Jun 22 08:56:10",
"host" => "centos7504",
"@version" => "1"
}
{
"syslog_severity_code" => 5,
"@metadata" => {
"ip_address" => "192.168.56.104"
},
"syslog_facility" => "user-level",
"syslog_facility_code" => 1,
"syslog_program" => "kernel",
"message" => "<6>Jun 22 08:56:10 server75-104 kernel: device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com",
"type" => "syslog",
"syslog_message" => "device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com",
"syslog_severity" => "notice",
"received_from" => "centos7504",
"@timestamp" => 2018-06-22T06:56:10.000Z,
"syslog_hostname" => "server75-104",
"port" => 58498,
"received_at" => "2018-06-22T06:56:59.191Z",
"syslog_timestamp" => "Jun 22 08:56:10",
"host" => "centos7504",
"@version" => "1"
}
I checked out other similar post on this forum, but until now I didn't find a solution. 
Someone can help me please? 
-Thanks