Logstash Configuration File Ordering Does Matter

Elastic Stack Logstash
1

Hi all,
I'm running logstash with a couple of -f switches and for some unknown reason, the order of the files does matter:

input-filter-output:
~/util/logstash-1.5.4/bin/logstash -f ./conf.d/input-50-email-sending.conf -f ./conf.d/filter-50-email-sending.conf -f ./conf.d/output-50-receiver.conf
Logstash startup completed
Logstash shutdown completed
 
input-output-filter:
~/util/logstash-1.5.4/bin/logstash -f ./conf.d/input-50-email-sending.conf -f ./conf.d/output-50-receiver.conf -f ./conf.d/filter-50-email-sending.conf
Logstash startup completed
Logstash shutdown completed
 
filter-output-input:
~/util/logstash-1.5.4/bin/logstash -f ./conf.d/filter-50-email-sending.conf -f ./conf.d/output-50-receiver.conf -f ./conf.d/input-50-email-sending.conf
Logstash startup completed
#gave is some time to run...
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed

Can someone explain this behaviour ?

Thanks,
Yarden

2

Huh. Very odd. I've seen a couple of other folks reporting the same symptom, i.e. that Logstash immediately terminates right away without anything interesting in the log. Do you get anything useful if you crank up the log level with --debug?

3

Hi Magnus,
With --debug, I found the following:

input-filter-output:

Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"http", :path=>"logstash/outputs/http", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}

input-output-filter

Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"csv", :path=>"logstash/filters/csv", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}

filter-output-input: Logstash starts and stays up:

Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}

I didn't expect seeing errors in the 3rd bullet (filter-output-input), but Logstash stays up.

If the full log is needed, I can pastebin it, but it will take me some time to sanitize it.

Thanks

4

Another interesting thing is that if I start Logstash with "bin/logstash - /etc/logstash/conf.d" (passing a directory containing my configuration) its working.

What is different between specifying specific files in certain order and specifying a directory containing the configuration?

5

What is different between specifying specific files in certain order and specifying a directory containing the configuration?

Logstash sorts the configuration files found in a directory by name before reading them. Given how you have named your files, perhaps they happen to end up in the order that happens to work?

6

My config files are standardised as follows:
filter-[ORDERING_NUMBER]-[DESC].conf => filter-50-CDN.conf
input-[ORDERING_NUMBER]-[DESC].conf => input-50-sockets.conf
output-[ORDERING_NUMBER]-[DESC].conf => output-50-http.conf

But we don't know that for sure, and if so, we don't know order that files should be ordered in.

Should I open an issue on main Logstash?

7

Yes, please open an issue for this. Configuration file ordering shouldn't matter except for the internal order of filters.

8

Issue opened.

Thank you,
Yarden

9

Hi,

can you tell me the order of letters and numbers?

lets say I have these files.

001-conf.conf
999-conf.conf
aaa-conf.conf
zzz-conf.conf

will this be run in the order above, or are letters used first and then numbers?

10

Standard ASCII ordering (for lack of a better term) is used, so numbers come first, then lowercase letters, then uppercase letters.

11

thanks for the fast response, that helps a lot

12

How can you do the same on Windows ? Because there is no conf.d folder to store config files.
Thanks!

13

How can you do the same on Windows ? Because there is no conf.d folder to store config files.

Just create one somewhere. Logstash will read all config files in any directory it's configured to use with -f.