Rsyslog to logstash help

[ajhstn]

Hello,

I am seeking advice regarding getting multiple syslog devices into logstash/elasticsearch.

I currently send syslog messages from multiple devices to a central rsyslog server on port 514. Because the messages are coming from different devices, different vendors, the message structure is different. I currently forward all messages from rsyslog to logstash on port 10514, using the *.*@ip-address-logstash-server; configuration.

My question is where/how should i best filter these messages before sending them from logstash to elasticsearch.

1. Does it make sense to use conditionals in my logatash filter based off the syslog [program] field and add a tag, then use another conditional in my logstash output to create an index for each syslog type/tag?
2. Am i correct in thinking all different types of input, should have its own output index?
3. Does it make sense to somehow do some filtering on the rsyslog side, and add some kind of tagging fields there in the message before sending it to logstash?

This is my architecture currently.

[warkolm (Mark Walkom)]

1. Yes
2. Depends, how different are they?
3. It may make things easier, but if the program value is unique then it's probably not worth the extra work.

[ajhstn]

Ta, can you please go a little deeper into your insight on "depends on how different they are"?

[warkolm (Mark Walkom)]

Data sources that are dramatically different should go into different indices.
But say you had an application that had a few different versions deployed in your environment, and they had slightly different formats (eg additional fields in newer versions), then adding them into the same index would be ok.