Variable in HTTP plugin

Elastic Stack Logstash
1

Hello,

I have a pretty basic use case for which I am unable to use log stash as intended and would appreciate community's help.

There are 'n' keys that are part of a URL. These keys keep changing on daily basis and I would like to loop through 'n URLs ' formed from those keys and store the data in index.

Below is my conf file where I am using 1 hardcoded key.

input
{
  http_poller  
    { 
		urls => 
		{
			logs => 
			{
				method => get
				url => "https://someURL/Hardcoded Key value"
				headers => {
					accept => "application/json"
				}
				           
			}
		
		}
		codec => "json_lines"
		schedule => { every => "1m"}
                               
	}
}


filter 
{
	json 
	{
		source => "message"
		some filtering here
	}
	
}

output 
{
	elasticsearch 
	{
		hosts => ["http://localhost:9200"]
		index => "MyIndex"
		user => "UN"
		password => "PW"
	}
}

I am looking for a way to use variable for the "key" component in URL and call this script 'n' times.

I am open to other suggestions as well.

2

Are you saying you want the url that the http_poller calls to be variable? I do not think that is possible.

3

Yes, a part of the URL needs to come from a variable.

Ex:https://test.sample.com/Logs?$filter=((JobKey eq <<JOBKEY>>&$orderby=TimeStamp desc

I want to pass JOBKEY as variable. (There are multiple job keys and they keep changing every 24 hours.)

4

You can reference an environment variable, but if the environment changes then logstash will only see the new value if it restarts.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.