Running multiple independent logstash config files with input,filter and output

Elastic Stack Logstash
1

Hi,

We have multiple applications running on a single server and we are collecting the logs from these application and sending it to a intermediate queues using logstash forwarders.

As, the log formats, processing and the output queues were different for different applications logs, I prepared multiple config files which are complete in themselves i.e. each config file has the input, filter and output section.

what is the best way to start all the configs,

  1. logstash -f /configDirPath is giving jumbling up the logs, i.e. it is sending log of one type to the queue of another.
  2. logstash -f /configDirPath/* is only starting the first file in the directory.

Is the only option in this case is to start multiple daemon threads as that is the only option working.

or am i missing something.

Again, each config file is complete in itself i.e. with input, filter and output section and the processing logic in all these config is also different so am not able to combine them as that is just making the config file not readable.

1 Like
2

Logstash currently has a single event pipeline. All configuration files are just concatenated (in order) as if you had written a single flat file. If you don't want all filters to apply to all events you need to have conditionals to select which filters (and outputs) to apply where. For example, you'd typically assign different types to different kinds of messages so you'd wrap your filters like this:

if [type] == "sometype" {
  ...
}
3 Likes
3

thanks Magnus,

but I am trying to avoid such a situation as if there is a scenario like we are changing the processing logic of a log or adding additional stuff then we need to stop the entire pipeline, instead of just taking one particular log off and not affecting the other ones.

So it seems, running multiple instance of logstash is the only way.

4

Yes, if you really need multiple event pipelines that can be stopped and restarted individually you will indeed have to run multiple instances of Logstash.

5

Hello

I've met the same problems as you describe, and i didnt understood at this moment why I had multiple times the same input.
I've decided to modify the init.d script to allow standalone execution of the script, with only one invocation.

http://pastebin.com/Z203qn1Q

The main asset is I dont have to set up type test all the time, the logs and errors are separated

Regards.

7

@vasumathy, please stop posting the same question in old unrelated threads.

9

@magnusbaeck any issues running multiple instances of Logstash with regards to ES bulk loading or better to use conditional logic and multiple configs on conf.d/ and let a single instance handle throttling ES load?