Multiple config files as logstash service

Elastic Stack Logstash
1

I am using logstash 2.1 version, my /etc/logstash/conf.d/ directory consist of multiple configuration like below.

unix-shipper.conf, firewall-shipper.conf, windows-shipper.conf,

When i enabled logstash as service, configuration files are not giving the output as expected, where as when i ran as process separately those are working fine.

Please let me know if we run logstash as service do we need to alter the /etc/init.d/logstash file, if yes what are all the changes?

2

Even if you use multiple configuration files they are effectively treated as a single large file. Logstash has a single event pipeline where events from all inputs are passed to all filters and outputs unless you use conditionals to restrict which filters and outputs receive which events.

3

Yes true, but my configuration files are sending output to Redis then Logstash indexer( unix-indexer.conf, windows-indexer.conf).

These are developed to recieve the events based on key value type (these types are different for each log source eg: type:unix, type:windows) .

When i run as service all events are marked as same event type (For example windows, vpn events are going to be marked as unix event type).

4

It'll be much easier if you show your configuration files instead of describing them.

5

My two configurations( One is for shipper and other for Indexer) for each logsource looks like below,
unix-shipper.conf

input
 { file         { path => ["/var/log/secure", "/var/log/messages"] type => "unix"  } }}
filter {}
output {
redis    {   host => [ "host1:7000", "host1:7001", "host2:7002", "host2:7003", "host3:7004", "host3:7005" ]
                        data_type => "list"  key => "unix"                         } }}

Unix-Indexer.conf

input {
redis {  host => "host1" port => "7000" data_type => "list" key => "unix" }
redis {  host => "host1" port => "7001" data_type => "list" key => "unix" } 
redis {  host => "host2" port => "7002" data_type => "list" key => "unix" }
redis {  host => "host2" port => "7003" data_type => "list" key => "unix" }
redis {  host => "host3" port => "7004" data_type => "list" key => "unix" }
redis {  host => "host3" port => "7005" data_type => "list" key => "unix" }
}


output {
elasticsearch { hosts => ["ESnode1:9200", "ESnode2:9200", "ESnode3:9200"]  }
email { } 
}
6

Right. And having these configuration files is equivalent to this:

input {
file         { path => ["/var/log/secure", "/var/log/messages"] type => "unix"  } }
redis {  host => "host1" port => "7000" data_type => "list" key => "unix" }
redis {  host => "host1" port => "7001" data_type => "list" key => "unix" } 
redis {  host => "host2" port => "7002" data_type => "list" key => "unix" }
redis {  host => "host2" port => "7003" data_type => "list" key => "unix" }
redis {  host => "host3" port => "7004" data_type => "list" key => "unix" }
redis {  host => "host3" port => "7005" data_type => "list" key => "unix" }
}
output {
redis    {   host => [ "host1:7000", "host1:7001", "host2:7002", "host2:7003", "host3:7004", "host3:7005" ]
                        data_type => "list"  key => "unix"                         } }
elasticsearch { hosts => ["ESnode1:9200", "ESnode2:9200", "ESnode3:9200"]  }
email { } 
}

So all events from the file and redis will be sent to the redis, elasticsearch, and email outputs. As I said, if this isn't what you want you need to use conditionals. However, in your case I'd run multiple Logstash instances.

7

Magnus,

unix-shipper.conf and unix-indexer.conf files are already at separate servers, all shipper configuration files are getting marked with same type(eg: type:unix even for windows and vpn )

8

And how do Windows and VPN events get into Logstash?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.