# Dynamic urls for Logstash http\_poller plugin **URL:** \ **Category:** Logstash\ **Created:** [December 29, 2020, 9:03am UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785 "2020-12-29T09:03:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![malsioufi](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@malsioufi](https://discuss.elastic.co/u/malsioufi)\ **Post date:** [December 29, 2020, 9:03am UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/1 "2020-12-29T09:03:59Z")
Hi, I have an implementation of some script that calls an api to start some job, I can follow up the job status another api (jobs-api). I want to use Logstash and Elasticsearch to monitor the status of the jobs I start, in order to do that I am using http\_poller input plugin. ```auto input { http_poller { urls => { test1 => "http://somehost/jobs?jobsLabel=nameOfApp_20201228" } request_timeout => 60 # Supports "cron", "every", "at" and "in" schedules by rufus scheduler schedule => { cron => "0 * * * * UTC"} codec => "json" # A hash of request metadata info (timing, response headers, etc.) will be sent here metadata_target => "http_poller_metadata" } } ``` As you can see in the urls part the endpoint follows the pattern: ```auto http://somehost/jobs?jobsLabel=sometext_yyyyMMdd ``` Where `nameOfApp_20201228` is some label my script creates when it starts the job, and it contains the name of the app followed by the date of the day when the job has been created in this example it is `20201228`. Is there a way to write the config file of logstash that dynamically updates the jobsLabel to be today's date? or at least to watch all job-labels starts with `nameOfApp_`? something like: ```auto urls => { test1 => "http://somehost/jobs?jobsLabel=nameOfApp_*" } ``` or ```auto urls => { test1 => "http://somehost/jobs?jobsLabel=nameOfApp_%{today}" } ``` Note: I am runing Logstash and Elasticsearch with Docker ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [December 29, 2020, 10:02am UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/2 "2020-12-29T10:02:04Z")
May be you can set [environment variable](https://www.elastic.co/guide/en/logstash/current/environment-variables.html) for that ```auto urls => { test1 => "http://somehost/jobs?jobsLabel=nameOfApp_%${today_AppName}" } ``` And find a way to upadte that env with a ruby filter, something like this may work ```auto ruby { code => ' ENV["today_AppName"] = "nameOfApp_" + Time.now.strftime("%d/%m/%Y %H:%M"); ' } ``` ---
**Author:** ![malsioufi](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@malsioufi](https://discuss.elastic.co/u/malsioufi)\ **Post date:** [December 29, 2020, 10:58am UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/3 "2020-12-29T10:58:45Z")
Thanks for your answer but I do not think I can set the environment variable in the filter since the filters are executed after the input plugins. ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [December 29, 2020, 11:48am UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/4 "2020-12-29T11:48:39Z")
I refered to this [thread](https://discuss.elastic.co/t/dynamic-input-http-poller-url/201928) Not sure if it's possible to update environment variable from ruby context ---
**Author:** ![malsioufi](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@malsioufi](https://discuss.elastic.co/u/malsioufi)\ **Post date:** [December 29, 2020, 11:59am UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/5 "2020-12-29T11:59:16Z")
I think this issue has been requested by many developers since long time but still no solid solution for it other than some work arounds ---
**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Post date:** [December 29, 2020, 12:02pm UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/6 "2020-12-29T12:02:33Z")
Yes, I'm one of them looking for that solid solution 🙂 for the moment i generate dynamically the configuration that it get loaded everytime updated. May be @Badger and @leandrojmp an advice on this topic ---
**Author:** ![malsioufi](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@malsioufi](https://discuss.elastic.co/u/malsioufi)\ **Post date:** [December 29, 2020, 12:17pm UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/7 "2020-12-29T12:17:49Z")
There is an open issue on GitHub for this problem since 2016 > > > As explained in Event Dependent Configuration > Some of the configuration options in Logstash require the existence of fields in order to function.... ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 29, 2020, 2:57pm UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/8 "2020-12-29T14:57:05Z")
> [@ylasri](#): > > i generate dynamically the configuration that it get loaded everytime updated I think that this is already the simplest solution for this case at the moment. I would use a simple script to change the `url` line every day, which would trigger a config reload. Or move the polling process to a external script and use the `exec` input plugin. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 26, 2021, 2:57pm UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785/9 "2021-01-26T14:57:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.