Logstash Configuration File Ordering Does Matter


(Yarden Bar) #1

Hi all,
I'm running logstash with a couple of -f switches and for some unknown reason, the order of the files does matter:

input-filter-output:
~/util/logstash-1.5.4/bin/logstash -f ./conf.d/input-50-email-sending.conf -f ./conf.d/filter-50-email-sending.conf -f ./conf.d/output-50-receiver.conf
Logstash startup completed
Logstash shutdown completed
 
input-output-filter:
~/util/logstash-1.5.4/bin/logstash -f ./conf.d/input-50-email-sending.conf -f ./conf.d/output-50-receiver.conf -f ./conf.d/filter-50-email-sending.conf
Logstash startup completed
Logstash shutdown completed
 
filter-output-input:
~/util/logstash-1.5.4/bin/logstash -f ./conf.d/filter-50-email-sending.conf -f ./conf.d/output-50-receiver.conf -f ./conf.d/input-50-email-sending.conf
Logstash startup completed
#gave is some time to run...
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed

Can someone explain this behaviour ?

Thanks,
Yarden


(Magnus Bäck) #2

Huh. Very odd. I've seen a couple of other folks reporting the same symptom, i.e. that Logstash immediately terminates right away without anything interesting in the log. Do you get anything useful if you crank up the log level with --debug?


(Yarden Bar) #3

Hi Magnus,
With --debug, I found the following:

input-filter-output:

Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"http", :path=>"logstash/outputs/http", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}

input-output-filter

Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"csv", :path=>"logstash/filters/csv", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}

filter-output-input: Logstash starts and stays up:

Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}

I didn't expect seeing errors in the 3rd bullet (filter-output-input), but Logstash stays up.

If the full log is needed, I can pastebin it, but it will take me some time to sanitize it.

Thanks


(Yarden Bar) #4

Another interesting thing is that if I start Logstash with "bin/logstash - /etc/logstash/conf.d" (passing a directory containing my configuration) its working.

What is different between specifying specific files in certain order and specifying a directory containing the configuration?


(Magnus Bäck) #5

What is different between specifying specific files in certain order and specifying a directory containing the configuration?

Logstash sorts the configuration files found in a directory by name before reading them. Given how you have named your files, perhaps they happen to end up in the order that happens to work?


(Yarden Bar) #6

My config files are standardised as follows:
filter-[ORDERING_NUMBER]-[DESC].conf => filter-50-CDN.conf
input-[ORDERING_NUMBER]-[DESC].conf => input-50-sockets.conf
output-[ORDERING_NUMBER]-[DESC].conf => output-50-http.conf

But we don't know that for sure, and if so, we don't know order that files should be ordered in.

Should I open an issue on main Logstash?


(Magnus Bäck) #7

Yes, please open an issue for this. Configuration file ordering shouldn't matter except for the internal order of filters.


(Yarden Bar) #8

Issue opened.

Thank you,
Yarden


(Luca Wintergerst) #9

Hi,

can you tell me the order of letters and numbers?

lets say I have these files.

001-conf.conf
999-conf.conf
aaa-conf.conf
zzz-conf.conf

will this be run in the order above, or are letters used first and then numbers?


(Magnus Bäck) #10

Standard ASCII ordering (for lack of a better term) is used, so numbers come first, then lowercase letters, then uppercase letters.


(Luca Wintergerst) #11

thanks for the fast response, that helps a lot


(Architha) #12

How can you do the same on Windows ? Because there is no conf.d folder to store config files.
Thanks!


(Magnus Bäck) #13

How can you do the same on Windows ? Because there is no conf.d folder to store config files.

Just create one somewhere. Logstash will read all config files in any directory it's configured to use with -f.


(system) #14