Same config file for test and development

Hi,
What I am trying to do is having the same logstash.conf file for my test and deployment/shipping. However for testing I want the output to go to my local elastic search, and for deployment to a http address.
I tried using an environment variable into a condition to check if the file is used for testing or not, and then redirect the output accordingly, but environment variables don't seem to be working inside condition.
I also tried using metadata with environment variables, see this example : https://github.com/elastic/logstash/issues/5115#issuecomment-214283550
but it did'nt work either.
Do you know if there is a solution for this or do I have to make 2 separate configuration files?

Thanks

I would do this by having a separate configuration files for just your output block.

output_test.conf

output {
     elasticsearch {
      ...define parameters elasticsearch
     }
}

output_deployment.conf

output {
     http {
      ...define parameters
     }
}

This way your test system can have just the test output configuration file and your production system can have your other output file. Just need to define this in the pipelines.yml

Thanks I'll give it a try. I never worked with pipelines.yml before, do you have an idea of what it would look like to define those .conf files?

You should read this article.

Essentially you can define any number of pipelines (flow) for your data to take.

Thanks for the article, I have a better understanding of pipelines now!
However I still don't understand how I can use them to define only the output block. They are running at the same time am I correct?
Because what I would need is that the data would go first into the common logstash.conf file (used for test and development) and the go to either of output files, depending on the situation. Is there a way to define that inside the pipelines.yml? Or maybe I should have 2 pipelines.yml file for each situation (test & deployment)?

Thanks for your time!

A pipeline will not start if the configuration file does not exist.
So on your test system you can have only the output_test.conf file and on your production system you can only have your output_production.conf file.
Or you can comment out the pipeline in the pipelines.yml file depending on the system.

Alternatively, can you put something in your incoming data that says

deployment.type:"test"
or
deployment.type:"prod" ?

Then you can include an if statement in your output to check if that field exists and output to the right place based on the incoming data.

I finally found a simpler way to do this. I stayed with your idea of having 2 output files, and then I used logstash *.conf to concatenate my common config file with either of the output files!
Thanks for your help!

1 Like