Logstash: Can we add http inputs dynamically in a filter section to config file from env variables

Elastic Stack Logstash
1

Hello,

Logstash: Can we add http inputs dynamically in a filter section to config file from env variables. Here is my requirement:

Below filter contains the URL to get Prometheus data for a server endpoint. Now I need to add a new server endpoint HTTP section under filter. Pls let me know if I can add dynamically new HTTP section to the config file from ENV variables without updating the filter manually? Kindly confirm if config changes auto-reload for filter section?

filter
{	
	http {
		url => "http://test.dev.com:11000/green/service/actuator/prometheus?tId=7667977e-6ddd-578a746b8812"
		verb => "GET"
		headers => 
		{
			"Authorization" => "Bearer %{token}"
		}
		target_body => "prometheusdataset"
	}
}
2

If logstash is running with auto-reload config enabled, then any changes in the config file will make it to be reloaded, so if you add a new filter block, this config will be reloaded.

But if you are using environment variables I'm not sure if a new variable will be picked up without a service restart or not, I think that environment variables are only loaded when the service starts.

3

Well, UNIX/linux world, there are extremely hacky ways to change the environment of an existing/running process (think attaching with gdb and/or poking around in kmem), dont even dream of going there. But yes, the process itself (usually) only reads the environment at early stages after starting up, so changing the env would not normally impact the process in any functional way.

So I can't see a way this is possible OOTB without restarting processes, which isn't too far from restarting the service (in the systemctl sense).

But you could certainly create another service that is dynamically modifying the logstash config files, which would then be auto-reload-ed, so has similar functional effect.

But @Bhanu_Praveen - respectfully - you are sort of suggesting a solution without really explaining anything around the problem you are facing. Maybe if you describe the actual problem, i.e. step back/up a level, then there might be a simpler approach.

4

Currently i have added one http end point inside the filter. There will more environments which i need to keep adding to this config file where i need to add multiple http sections in the config file. I do not want to update logstash file manually everytime.

Instead of adding the http section for every endpoint[to poll new environment], can i dynamically add the http sections in config file which will re-load automatically? So, no manual interaction at logstash level.

Hope you got my problem here. TIA

5

You can split your logstash configuration into multiple files and in pipelines.yml configure it to look at a directory, so every time a new file is added to this directory, the configuration is reloaded.

This is a pretty common way to manage pipeline configurations.

For example, in your pipelines.yml you would have something like this:

- pipeline.id: your-pipeline
  path.config: "/path/to/configurations/for/this/pipeline/*.conf"

Then in the specified path you would have multiple files with your configuration, a common approach is to have different files for inputs, outputs and filters.

I normally organize the files this way:

  • 000-input-something.conf
  • 999-output-something.conf

And for filters I start the files from 1XX until 8XX depending on the order that I want the filter to be processed, because the files will be merged into a big configuration in the order they appear when listing the path.

So, you could have multiple files with the http filters and if you want to add a new one, just put a new file in the path and logstash will reload the config.

The process of adding new files to the directory can be automated as well, there are a lot of tools to do that.

1 Like
6

You understand what I meant now I hope :slight_smile:

Excellent suggestions from @leandrojmp , as ever.

IMhO your actual real problem had not that much to do with "environment variables", rather more related to supporting multiple (and dynamic) environments.

My only addition to what @leandrojmp wrote is also try to consider what happens when environments need to be deleted too. People often forget that step. Dynamic cuts both ways.

1 Like
7

Yep, currently my pipelines folder contains the same way how you mentioned:

01-beatsinput.conf
02-gcinput.conf
03-beatsfilter.conf
09-elasticoutput.conf

I was not sure about the new file addition will auto reload configuration, because i read smewhere that all input files cannot be auto reloaded which interact with system.

Got it, i will have multiple filter files created and moved to pipeline folder.

Thanks

8

Not sure where you read this, but it does not seem correct.

Logstash will reload your configuration if it is configured to do that, per default it is not.

You need to add these lines into logstash.yml and restart the logstash service.

config.reload.automatic: true
config.reload.interval: 30s

This will make logstash check for changes every 30 seconds and reload the pipeline if anything has changed.

9

Then i am not sure what the below last section in this page means?

> ##
> Plugins that prevent automatic reloading
> 
> [edit](https://github.com/elastic/logstash/edit/8.17/docs/static/reloading-config.asciidoc)
> 
> Input and output plugins usually interact with OS resources. In some circumstances those resources can’t be released without a restart. For this reason some plugins can’t be simply updated and this prevents pipeline reload.
> 
> The [stdin input](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-stdin.html) plugin, for example, prevents reloading for these reasons.