How do I setup Logstash to segregate logs from multiple servers?

Elastic Stack Logstash
1

I building a central ELK stack that will be used to aggregate the logs from different customer installations, with each installation having multiple servers. I use rsyslog as the shipper. I wan to be able to "see" the following example structure in my ELK server

/path_to_logs/
    - Customer1
    ---- Server1
    ------ app-error.log
    ------ app-system.log
    - Customer2
    ---- Server1
    ------ app-error.log
    ------ app-system.log
    ---- Server2
    ------ app-error.log
    ------ app-system.log

I started reading about rsyslog templates but am not making the connection yet. For example, how can I embed the "Custoer name" and "Server name" information in the log, and then how can Logstash extract that information?

2

If rsyslog knows the customer name it should be easy to include it as (for example) a prefix of each message. Logstash can then extract the customer name along with the hostname, timestamp, and whatever else you've got.

3

Thanks @magnusbaeck. I guess my question is, how can I make rsyslog (from remote client) "know" the customer name? This is the mechanism I'm missing.

4

I figured I'll eventually figure this out.

The trick is in the use of rsyslog tempate, where one can store key/value pairs.

So I created an rsyslog list template, and have the following key/pairs, among othes

template(name="textLogTemplate" type="list") {
...
  constant(value="\"customer\":\"customer1\", ")
  constant(value="\"role\":\"app-server1\", ")
...
}

Then in my Logstash configuration files, I have the following

output {
  path => "/var/log/trm/%{customer}/%{role}/trm-system.log"
  codec => line { format => "%message" }
}

Which creates a /var/log/trm/customer1/app-server1/trm-system.log file and puts the value of %{message} in it.

Hope someone benefits from this.

1 Like