Logstash pipelines and conditionals in config

Elastic Stack Logstash
1

Hi All,

Although the subject sounds interesting (maybe for some) but I don't really have a clue what I'm doing! I've been trying to search for an answer but no luck so I'm seeing if someone can help or point me to something that makes sense!

I have a number of logstash config files (grok mutate etc) and they work perfectly fine when directly starting logstash with -c, additionally adding them to pipelines.yml works fine too and I can see them referenced in the logstash console.

Within my logstash config files I'm doing something like this (*not a real example just psudo code)

if [customtag] == "tagA" do something

and in a separate file the same thing but referencing a different servername

if [customtag] == "tagB" do something else...

Now if the tag is actually tagA I don't have any problems, however if it's not, it doesn't proceed onto the other config files to check if they match any of those defined there. Looking at the console, it looks like it drops the connection and restarts logstash (in some way) in a massive infinite loop, only if it doesn't work with the first defined config file.

My question is 2 part, firstly what's going on here and secondary how do I get Logstash to work either via:

a) Recurse all config files for the correct match, b) Somehow have a "check next config file" line without having to define all the config filea
c) anything else more sensible..

The custimtag field is added via filebeat depending on the directory the logs pickup from but seems to be a "I don't understand" problem rather than a technical fault.

Can anyone assist?

Thanks Ollie :slight_smile:

2

Hi ollie,

This is absolutely correct. Depending on the OS and method of installation logstash will concatenate all the config files and create a single file, depending on the naming convention of the config files. (Logstash Directory Layout | Logstash Reference [8.11] | Elastic)

It will then proceed to match events as per the associated filters. Mind you, a lot of if..else loops will slow down the pipeline. Try to use the multiple pipeline feature if you can. Multiple Pipelines | Logstash Reference [8.11] | Elastic

I personally use the exact same approach that you have described and it is working perfectly fine.

3

Ah okay cool doesn't look like i'm doing something crazy then which is good!

On closer inspection I'm getting a "Error: Address already in use: bind" from the Logstash terminal output so now I'm pretty sure you cant use a single beats port on the same host to multiple config files unless i spin up multiple filebeat instances... That's of course unless I need to remove something specific from the config file in Logstash to stop it spawning multiple ports for each config file?

4

If path.config points to a directory all the files in that directory are combined (including foo.conf, foo.conf-, foo.conf.bkup) into a single configuration. Events are read from all the inputs, put through all the filters (unless you are using conditionals), and sent to all the outputs. If one config file starts an input listening on a TCP or UDP port then no other file can do so.

5

Why not use a single Beats input plugin config and listen for the events on it?

Then you can use the tag based conditionals approach that you described to parse the data. Shouldn't be a problem.

6

Yeah you are 100% right but if I'm being honest I don't really know enough about the stack to justify installing filebeat on our production servers to harvest the logs.

At the moment I'm manually copying them off into a dev environment where logstash happily crunches though them, not quite real-time but it serves its purpose in this specific instance.

7

Thanks to both of you for your replys replys they made sense sense and I have since combined them into a single config file and it seems to work as expected, thanks for the response!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.