How to manage large config files

Elastic Stack Logstash
1

As I continue to test logstash I want to make it easier to read and edit. Meaning that my config file is getting t0o large because I am pulling logs of many types each having their own unique way of being parsed. So is there any ways to have parts of the filter point to another file to look for how to filter. The problem is mainly looking at many filter types...

1 Like
2

You can split your config file into any number of files, put them in a directory, and pass Logstash the path to that directory. The files will be read in alphabetical order. If you don't want all filters to apply to all messages you (still) need conditionals around the filters.

3

Are saying I can run many instances of Logstash? Trying to clarify.

Yes I want to apply filters to certain messages so I will keep conditionals around the filter. But how would I point logstash to those files in the config

i.e.

filter {
if [type] == blah
{
point to x filter file
} else if [type] == blahblah {
point to y filter file
}

Is there documentation on this process? looking through it right now. Will post link if I find it.

Thanks,
M

4

Are saying I can run many instances of Logstash?

You can run as many instances as you like, but above I was just saying that you can split your single configuration file into multiple files.

filter {
if [type] == blah
{
point to x filter file
} else if [type] == blahblah {
point to y filter file
}

No, you can't do that but you can do this:

x.conf:

filter {
  if [type] == "blah" {
    ...
  }
}

y.conf:

filter {
  if [type] == "blahblah" {
    ...
  }
}

That's equivalent to this:

xy.conf:

filter {
  if [type] == "blah" {
    ...
  }
  if [type] == "blahblah" {
    ...
  }
}
5

Yes was trying to avoid the running multiple instances because I think there are memory issues there when you try spawn more and more instances. The other method you mentioned seems right where I want to use but I want to understand the run down of logstash.

If I have x.conf with some filter and y.conf with another filter how would I run both on the same logstash instance? Meaning that:

every time I have been running a something.conf file I spawn another logstash with:
./bin/logstash -f something1.conf
./bin/logstash -f something2.conf

but in your case if i point to a directory as such
./bin/logstash -f *.conf

What is the difference? I ask because spawning more logstash instances uses up a lot of my memory, but maybe seeting many .conf files will limit that memory usage. I think this will clear up a lot of questions.

Thanks,
M

6

Yes was trying to avoid the running multiple instances because I think there are memory issues there when you try spawn more and more instances.

Sure, there's of course a resource overhead with multiple instances.

but in your case if i point to a directory as such
./bin/logstash -f *.conf

Here you're not passing a directory, you're passing a wildcard which will be expanded by your shell to e.g. ./bin/logstash -f x.conf y.conf (which Logstash won't like). Pass the path to the directory.

The standard RPM and Debian packages are set up to read all files in /etc/logstash/conf.d.

What is the difference?

You don't see the difference between starting multiple instances of Logstash that each read a single configuration file and starting a single instance of Logstash that reads multiple configuration files?

7

A few of my LS config files contains 500+ lines now, but before thinking about splitting them into separate files, I wonder if there are any performance impacts between the following filter blocks

Same config file

filter {
    if [type] == "type1" {
        do something
    } else if [type] == "type2" {
        do something
    } else if [type] == "type3" {
        do something
    } else if [type] == "type4" {
        do something
    }
}

Separate config files (note: we can't use else if in separate case, can we?)

filter {
    if [type] == "type1" {
        do something
    } 
    
    if [type] == "type2" {
        do something
    }
    
    if [type] == "type3" {
        do something
    } 
    
    if [type] == "type4" {
        do something
    }
}

If we have 15+ types and thousands of messages per second through Logstash (a separate queue is not an option now), will the performance be signficantly different between both cases? I tend to put the types with lots of messages on top of the filter block so they get out of the filter block faster.

8

The first example will perform better since Logstash won't have to perform quite as many string comparisons, but whether it's significant to you is another story. I suggest you measure what the impact is.

9

Got it, so I think when our message volume reaches certain levels (like 100,000+ msgs per second), there is a trade off between having better performance and a cleaner LS config. With low message volume, it doesn't really matter.