Rsyslog to logstash use mmsequence mod


(rockylee) #1

HI!

i want to use rsyslog mmsequence & omfwd send log to logstash
but nothing log send to logstash
please help me to fix error !
thank you

rsyslog os:centos 7 10.5.1.189
elk os:centos 7 10.5.1.111

rsyslog.conf

#rsyslog configuration file

For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

$template insertpl,"insert into SystemEvents (Message, Facility, FromHost, FromIP, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', '%fromhost-ip%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

MODULES

$ModLoad ommysql
.:ommysql:localhost,Syslog,rsyslog,XXXXXXX;insertpl

The imjournal module bellow is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
$ModLoad immark

Provides UDP syslog reception

#$ModLoad imudp
#$UDPServerRun 514
$ModLoad imudp
$UDPServerRun 514

Provides TCP syslog reception

$ModLoad imtcp
$InputTCPServerRun 514
$ModLoad omfwd
$Modload mmsequence
Ruleset( name="forwardRuleSet" ) {
Action ( type="mmsequence" mode="instance" from="0" to="4" var="$.seq" )
if $.seq == "0" then {
action (type="omfwd" Target="10.5.1.111" Port="5511" Protocol="tcp" queue.size="150000" queue.dequeuebatchsize="2000" )
}
if $.seq == "1" then {
action (type="omfwd" Target="10.5.1.111" Port="5512" Protocol="tcp" queue.size="150000" queue.dequeuebatchsize="2000" )
}
if $.seq == "2" then {
action (type="omfwd" Target="10.5.1.111" Port="5513" Protocol="tcp" queue.size="150000" queue.dequeuebatchsize="2000" )
}
if $.seq == "3" then {
action (type="omfwd" Target="10.5.1.111" Port="5514" Protocol="tcp" queue.size="150000" queue.dequeuebatchsize="2000" )
}
}

logstash.conf

input {
tcp { port => "5511" type => "syslog" }
tcp { port => "5512" type => "syslog" }
tcp { port => "5513" type => "syslog" }
tcp { port => "5514" type => "syslog" }
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
redis {
host => "localhost"
data_type => "list"
key => "logstash"
}
}


(Magnus B├Ąck) #2

Be systematic. Is rsyslog attempting to sending data to Logstash? Use tcpdump to confirm if you have to. Disable Logstash's redis output and use a plain stdout output to remove one error source and make sure you capture everything. Is there anything in the Logstash logs if you enable verbose logging with --verbose or --debug?


(system) #3