Issue with Multiple Logstash Conf files

Elastic Stack Logstash
1

I am facing a problem when multiple logstash conf are used within a folder and trying to connect the same using filebeat. I had posted this issue under filebeat section and now I am asked to post it under logstash as well.

Can anyone please let me know how to resolve this issue using multiple conf files ...

2

You can't have more than one Beats input plugin listening on the same port in more than one configuration file. Logstash concatenates the contents of all configuration files and keeping them separate is meant as an organizational aid for you. Logstash doesn't care about how you divide the input, output, and filter blocks into different files.

3

That means if I have multiple beats port configured under multiple logstash conf files, things should work ?

4

Yes, but why would you want to have multiple beats listeners?

5

I would like to have multiple conf files for multiple file path. Because I have multiple patterns and easy to maintain a file with few lines. Easy to understand and edit later.

6

Sure, but you can still have multiple configuration files for your filters even if you have a single beats input (that you can put in a file of its own if you like).

7

But in that case only my alphabetically ordered 1st conf is capturing logs.The other ones are ignored or it is not capturing anything.

8

So have you removed one of the beats inputs then? What do your configuration files contain now?

9

You mean a single prospector in filebeat ?

10

No, I mean that you should have a single beats input plugin on the Logstash side. The number of prospectors doesn't matter.

11

I have only 1 beat port configure.

Here is my FB conf files. -

  • input_type: log
    paths:
    • D:\ServiceLogs\Zephyr**
    • D:\Zephyr\inetpub\LogFiles**
      tags: ["Zephyr"]
      ignore_older: 1h
  • input_type: log
    paths:
    • D:\ServiceLogs\Nightingale**
    • D:\Nightingale\inetpub\LogFiles**
      tags: ["IVR"]
      ignore_older: 1h
  • input_type: log
    paths:
    • D:\FCServices\inetpub\LogFiles**
      tags: ["FCServices"]
      ignore_older: 1h
  • input_type: log
    paths:
    • D:\BreezeServices\inetpub\LogFiles**
      tags: ["BreezeServices"]
      ignore_older: 1h

Logstash 1 conf -

input
{
beats
{
port => 5044
}
}

filter
{
kv
{
value_split => ":"
remove_char_key => "[]"
remove_char_value => "[]"
include_keys => [ "method", "reasonPhrase", "requestUri", "content", "Payload", "id", "ClientID" ]
recursive => "true"
}

if "Zephyr" in [tags]
{
	mutate
	{
		replace => { "type" => "Zephyr" }
	}
}

if "IVR" in [tags]
{
	mutate
	{
		replace => { "type" => "IVR" }
	}
}

mutate
{
	add_field => { "LogType" => "Services" }
	remove_field => [ "tags", "offset", "input_type", "beat" ]
}

}

output
{
if [type] == "Zephyr"
{
elasticsearch
{
index => "zephyr-%{+YYYY.MM.dd}"
hosts => ["server:9200"]
}
}

if [type] == "IVR" 
{
	elasticsearch 
	{
		index => "ivr-%{+YYYY.MM.dd}"
		hosts => ["server:9200"]
	}
}
#stdout
#{ 
#	codec => rubydebug 
#}

}

Logstash 2 conf -

input
{
beats
{
port => 5044
}
}

filter
{
#Ignore log comments
if [message] =~ "^#"
{
drop {}
}

grok 
{ 
	match => 
	{
		"message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:HttpVerb} %{URIPATH:RequestUri} %{NOTSPACE:querystring} %{NUMBER:Port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:username} %{NUMBER:ResponseCode} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}"
	}
}

#Set the Event Timestamp from the log
date 
{
	match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
	timezone => "Etc/UTC"
}

#Ignore all values for which GROK pattern is not set at this moment.
if "_grokparsefailure" in [tags]
{
	drop {}
}

if "Zephyr" in [tags]
{
	mutate
	{
		replace => { "type" => "Zephyr" }
	}
}

if "IVR" in [tags]
{
	mutate
	{
		replace => { "type" => "IVR" }
	}
}

if "FCServices" in [tags]
{
	mutate
	{
		replace => { "type" => "FCServices" }
	}
}

if "BreezeServices" in [tags]
{
	mutate
	{
		replace => { "type" => "BreezeServices" }
	}
}

mutate
{
	add_field => { "LogType" => "IIS" }
	remove_field => [ "@version", "log_timestamp", "site", "querystring", "username", "clienthost", "useragent", "subresponse", "scstatus", "tags", "offset", "input_type", "beat" ]
}

}

output
{
if [type] == "Zephyr"
{
elasticsearch
{
index => "zephyr-%{+YYYY.MM.dd}"
hosts => ["server:9200"]
}
}

if [type] == "IVR"
{
	elasticsearch 
	{
		index => "ivr-%{+YYYY.MM.dd}"
		hosts => ["server:9200"]
	}
}

if [type] == "FCServices"
{
	elasticsearch 
	{
		index => "fcservices-%{+YYYY.MM.dd}"
		hosts => ["server:9200"]
	}
}

if [type] == "BreezeServices"
{
	elasticsearch 
	{
		index => "breezeservices-%{+YYYY.MM.dd}"
		hosts => ["server:9200"]
	}
}
#stdout
#{ 
#	codec => rubydebug 
#}

}

12

I have only 1 beat port configure.

No, you have one in each configuration file. You can only have one in total (listening on the same port).

13

So in my case what would be the ideal solution that you can suggest.

I wanted to keep these conf sepearte for future maintenance and easy understanding.

14

You can organize your configuration in any way you want as long as you make sure you don't have more than one beats input listening on the same port. You could e.g. have one configuration file containing all inputs, multiple files containing filters for various purposes, and one file with output plugins.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.