How to forward logs from multiple folder storage locations of rsyslog server to ELK?

[vignesh.s2 (vignesh)]

My case is, I have ELK setup were rsyslog forwards the logs to it.

My templates are ,

$template templmesg,"/data01/RemoteLogs/DLF/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

$template mylogsec,"/data01/RemoteLogs/Logserver/%$YEAR%/%$MONTH%/%HOSTNAME%/%HOSTNAME%-%$DAY%-%$MONTH%-%$YEAR%.log"

if $fromhost startswith "10.100.10" then ?templmesg

& stop

if $fromhost startswith "10.100.112" then ?mylogsec

& stop

so I have two locations were logs are stored

Because of multiple locations of log storages like DLF and Logserver. Kibana from (ELK) does not show logs which is received from rsyslog. It only reads from one location of logs that is from DLF/ dir and not from Logserver/ dir

Now am stuck and dont know "How to forward logs from multiple folder storage locations of rsyslog server to ELK?" or "is there any specific configuration in rsyslog that i need to work out?"

[vignesh.s2 (vignesh)]

SOLUTION and ITS WORKING

Since the rsyslog.conf configuration file is parsed from top to bottom, the actions are carried out in sequence for each message in the same order as they are defined in the file. What happens in your case is that the messages matching the $fromhost startswith "10.100.112" test are processed (i.e. written to the log files specified by the 'mylogsec' template) and then discarded by the stop statement.

The solution to this problem is straightforward. Before the message is dropped by rsyslog, you have to forward it to the remote Logstash server. You can modify your filter as shown below:

if $fromhost startswith "10.100.112" then ?mylogsec 
& @10.100.10.30:10514;json-template
& stop

Because you're using the JSON template for forwarding here, you will also need to move the definition of that template before the filter expression. So the final structure will be the following:

$template templmesg...
$template mylogsec...

template(name="json-template"...

if $fromhost startswith "10.100.112" then ?mylogsec 
& @10.100.10.30:10514;json-template
& stop

Once done, restart your rsyslog daemon for the changes to take effect.