Running logstash(pipeline.conf) through terminal by passing environment variables

[vaseemQA (Vaseem Akram Mohammad)]

Hi Team,

I need some help in running my Logstash project files.
I've a list of .conf files in a project called pilotProject, currently I test them from my local by running following command
./logstash -f ~/pilotProject/es.message.processing.conf --path.data ~/pilotProject/ and this runs es.message.processing.conf successfully and the problem for me is I hardcoded all the required variable values of es.message.processing.conf and it works perfectly.
PROBLEM: we are maintaining ~/pilotProject/environment.sh file and passing all the required variable values to .conf via environment.sh.
So I want to run .conf along with .sh in my local by passing the environment variables so That I can ensure whether .conf is properly resolving all the environment variables successfully through environment.sh

Thanks,
Vaseem

[adrianfusco (Adrian)]

Hello,

Did you take a look into this? Using environment variables in the configuration | Logstash Reference [8.2] | Elastic

[vaseemQA (Vaseem Akram Mohammad)]

Hey @adrianfusco

Thanks for the reply, Yeah I know This but my project implementation is not in this way, So I've to adopt the way it is
I just want to know how can we run environment.sh along with *.conf file

for Example : see below
example.conf is like below

input{
      log_group => <%= test_group %>
}

in the same directory I've created a file called envionment.sh and it looks like below

PROD_LOG_GROUP = `["/aws/test/log"]`

for file in *.conf ; do
 
  if [[ $file =~ "example.conf" ]]
  then 
    if [[ $ENVIRONMENT == "prod" ]]
    then 
     sed -i "s#<%= test_group %>#$PROD_LOG_GROUP#g" $file
   fi
fi
done

Now I want to run these two files from my local instead of deploying on the respective environment, so that i can verify whether <%= test_group %> from example.conf have "/aws/test/log" or not.

Thanks,
Vaseem

[leandrojmp (Leandro Pereira)]

Can you give an example of what you want to do? It is not clear what you want to do and what is not working.

From what you shared you are using a place holder in your configuration file and then you use an external tool (the environment.sh script) to change the configuration file to use an environment variable.

It is not clear what is the issue with Logstash as this is a custom automation.

To use an environment variable you set it in the config and then export the variable in a way that the user running logstash can access it.

Also, the correct way to refer to an environment variable in the config is ${VARIABLE}, not $VARIABLE.

[vaseemQA (Vaseem Akram Mohammad)]

Hi @leandrojmp

Thankyou so much for taking the time to help me,

So let me put this way, If I make any change in example.conf file I test it locally by hardcoding the environment variable values and Then I revert those hardcoded variable values and then deploy to respective environments like, qa, dev or prod, based on deployed environment all the variables get resolved via environment.sh file , so Now I want to Test the same in my local

Example: I run logstash from my local by hardcoding the values in example.conf file without reading them from environment.sh using this command
./logstash -f ~/pilotProject/es.message.processing.conf --path.data ~/pilotProject/

And What I'm looking for is:

I don't want to hardcode the variable values in example.conf instead I want them to pass through environment.sh file, to achieve this how should I run example.conf and environment.sh from my local machine? what should be my command to run from the local terminal?

Thanks in Advance,
Vaseem | +91 90300-25552

[leandrojmp (Leandro Pereira)]

There is nothing in logstash to do that, if you want to use environment variables in a Logstash configuration file, you need to set it as ${VARIABLE} in the config file.

What you want is to build a custom automation, which is out of the scope of this forum.

You need to build a shell script to change the values in your config file and the you can logstash with those values in the config file, maybe write a shell script that changes the variables in the config file, as the example you shared, and the same script would call logstash.