How to setup multiple config file in logstash

[malleshrp (Malleswararao)]

I want to setup multiple config files in logstash..
currently running individual .conf files which will transfer data from mysql to elasticsearch.
I am having 90 files how do I setup in one config file for production server.

Please provide suggestions.

[ITIC (Jordi)]

Hi

If I understand correctly, you want to have multiple .conf files in a single logstash pipeliine, each with its own input{}, filter{} and output{} sections.

To do that, you can put all your .conf files in the pipeline directory (as defined in your pipelines.yml) and use unique tags in each and every one of them, like this:

input {
  jdbc {
    tags => "Unique tag for .conf file #n"
    id => "jdbc_Unique ID for .conf file #n"
    
    jdbc_driver_class => [...]
    
  } # End jdbc {}
} # End input {}

filter {
  if "Unique tag for .conf file #n" in [tags] {
    
	<your filters here>
    
  } # End Unique tag for .conf file #n
} # End filter {}

output {
  if "Unique tag for .conf file #n" in [tags] {
    
	stdout {
    }
    
	<your outputs here>
	
  } # End Unique tag for .conf file #n
} # End output {}

In your case #n goes from 1 to 90. So yes, you'll have to edit all of them, sorry.

Hope this helps

[Fabio-sama (Fabio)]

You either want something similar to what @ITIC suggested, or you simply want to run the logstash instance once and have all your conf files be run.

If that's the case, simply specify various pipelines in the config/pipelines.yml file like the following:

- pipeline.id: my_first_pipeline
  path.config: "/path_to_first_pipeline.conf"
- pipeline.id: my_second_pipeline
  path.config: "/path_to_second_pipeline.conf"

And then simply run logstash without any additional option (like bin/logstash from the logstash directory). It'll run all the pipelines specified in the pipelines.yml file.

Obviously, you can add specific options for each individual pipeline in the pipelines.yml file (e.g. number of workers, batch_size, etc..).