Multiple filter files for different host

Elastic Stack Logstash
1

Hi ,

I have installed ELK stack on a linux machine and sending events using filebeat from different files.I have installed filebeat on different machines.

The process is like below

filebeat => logstash shipper => redis => elastic search

I have logstash shipper on centralized server for receiving log events from filebeat. filebeat send data to logstash shipper on the port : 5044 and shipper is running on the port.

I need to set up different configuration files of shipper for the machines sending data from filebeat. for example machine 1 should have machine1 shipper configuration.

Is it possible to use multiple logstash configuration for different machines using one single port or is there an alternative method for it.

Am getting port conflict for it when i have tried it,

I have also tried to set up multiple filter configuration file for different machines with one single input and output,but it does not seems to be working. The logs are not parsing via the filter.

anybody have a solution to this?.

2

Use conditionals based on variables, like the hostname - https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals

3

Hi Warkolm,

I have used the conditional hostname and created two filter files inside the logstash configuration directory.

but for some reason logstash does not working with two filter files at a time.

filter configuration for one machine

filter
{
if [host] == "datanode2"
{

if [type] == "log"
{
grok
{
match => [ "message", "%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: %{WORD:authentication}" ]
add_tag => [ "log", "grokked" ]
}
}

if [type] == "log" and [authentication] not in ["Failed","info"]
{
drop{}
}

if [type] == "iisbeat"
{
#ignore log comments
if [message] =~ "^#" {
drop {}
}

grok {
# check that fields match your IIS log settings
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostip} %{WORD:method} %{NOTSPACE:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
}
}

#Set the Event Timesteamp from the log
date {
match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
timezone => "Etc/UTC"
}

useragent {
source=> "useragent"
prefix=> "browser"
}

mutate {
remove_field => [ "log_timestamp"]
}
if [type] == "iisbeat" and [response] not in ["404", "444"] {
drop {}
}

if [type] == "wineventlog"
{

if [event_id] not in [4625,4666] {
drop {}
}
}

if [type] == "apachebeat"
{

grok {
match => { "message" => "%{COMMONAPACHELOG}" }
add_tag => [ "apachebeat", "grokked" ]
}

}
if [type] == "apachebeat" and [response] not in ["404", "403"] {
drop {}
}

}
else if [tags] not in ["grokked"]
{drop{}}
}

filter configuration for another machines

filter
{
if [host] == "datanode3"
{
if [type] == "log"
{
grok
{
match => [ "message", "%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: %{WORD:authentication}" ]
add_tag => [ "log", "grokked" ]
}
}

if [type] == "log" and [authentication] not in ["Failed","info"]
{
drop{}
}

if [type] == "iisbeat"
{
#ignore log comments
if [message] =~ "^#" {
drop {}
}

grok {
# check that fields match your IIS log settings
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostip} %{WORD:method} %{NOTSPACE:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
}
}

#Set the Event Timesteamp from the log
date {
match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
timezone => "Etc/UTC"
}

useragent {
source=> "useragent"
prefix=> "browser"
}

mutate {
remove_field => [ "log_timestamp"]
}
if [type] == "iisbeat" and [response] not in ["404", "444"] {
drop {}
}

if [type] == "wineventlog"
{

if [event_id] not in [4625,4666] {
drop {}
}
}

if [type] == "apachebeat"
{

grok {
match => { "message" => "%{COMMONAPACHELOG}" }
add_tag => [ "apachebeat", "grokked" ]
}

}
if [type] == "apachebeat" and [response] not in ["404", "403"] {
drop {}
}

}
else if [tags] not in ["grokked"]
{drop{}}

}

4

I have seperate input and output files inside the logstash conf directory.