General Config Question

Elastic Stack Logstash
1

I have multiple logstash config files in my conf.d dir. My question is two fold:

  1. I see many people naming their config s with a preceding number. Is this because logstash will load them in numerical order?
  2. Should I organize my conf files as input.conf, filter.conf, and output.conf? I filter multiple types of records from many sources and I fear that one big filter conf will get hard to manage not to mention hard to figure out what is getting parsed and when.

I want to know what others are doing in production. Right now in a fairly large deployment I have each type of system going to its own conf file. Network sensors to their own conf, Database servers to theirs, web servers to theirs, etc. Most of my logstash nodes send to a central queue which then a set of logstash nodes read from and begin processing with filters, mutates, grok, geo, etc. then they get indexed into ES.

I just want to make sure things are being done as efficiently as possible and the way the developers intended it to be done.

Thanks for the help

2

Logstash does indeed load the files in numerical order.

I adopted this approach that seems to work well:

100-input.conf
300-db-filter.conf
301-network-filter.conf
900-output.conf

I separate each type into it's own filter config.

One thing to be aware of if you follow this approach is that all of the files in the conf.d directory will be read and acted upon by Logstash, so each of your individual files must contain a condition that limits the filter actions to that particular type, eg:

if [type] == "Network" {
     run network filters
}

I accidentally neglected to do this on one of my filters and ended up applying unnecessary tags to some of my inputs.

1 Like