Logstash gives error while trying to start multiple config files

Elastic Stack Logstash
#1

Hi,

We are collecting the application logs generated on one server, sending them to logstash using Filebeat. There are different format/types of log data, we had individual logstash config file to parse the data( i.e. each config file has the input, filter and output section). For all the config files, Filebeat will be the input.

So, i have copied all the logstash config files in to one single directory (config directory). When i started the logstash, it gives the following error:

$ bin/logstash -f ConfigDirPath/

Pipeline aborted due to error {:exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:ininitialize'", "org/jruby/RubyIO.java:853:in new'", "/home/custom/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/lumberjack/beats/server.rb:51:ininitialize'",`

Is there any option to run all the config files at the same time to read events sent by filebeat ?

Again, all the config files has the following in their input section:
input {
beats {
port => 5044
}
}

#2

It will do that by default if it is pointed to a directory.

Maybe you have a duplicated config?

#3

We are collecting the application logs generated on one server, sending them to logstash using Filebeat. There are different format/types of log data, we had individual logstash config file to parse the data( i.e. each config file has the input, filter and output section). For all the config files, Filebeat will be the input.

That's not how Logstash works with multiple configuration files. The files won't be independent in the way you think. Logstash has a single event pipeline where filters from all configuration files process events from inputs in all configuration files and then handing off the results to outputs from all configuration files. You can wrap filters and outputs in conditionals to avoid having them apply to every single events.

Again, all the config files has the following in their input section:
input {
beats {
port => 5044
}
}

Because of the reasons stated above this won't work and will predictably result in exactly the error you're getting.

#4

I have "beats plugin" in the input section for all the config files (3 files). That is the only duplicate and common config in all config files.

#5

That is the only duplicate and common config in all config files.

Yes, and such duplication won't work.

#6

@magnusbaeck Thanks for your response. So, all the config files are using filebeat in the input section. While we start logstash it uses the first config file from the available files, when it's trying to load second config file from the list it's giving error because filebeat is already in use with the first config file.

Is there any option to have common input file for all config files.

#7

List the beats input exactly one time in one of the files (or a completely separate file). If messages from different hosts or of different types should be filtered differently you can use conditionals.

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

#8