Dynamic urls for Logstash http_poller plugin

Elastic Stack Logstash
#1

Hi, I have an implementation of some script that calls an api to start some job, I can follow up the job status another api (jobs-api). I want to use Logstash and Elasticsearch to monitor the status of the jobs I start, in order to do that I am using http_poller input plugin.

input {
  http_poller {
    urls => {
      test1 => "http://somehost/jobs?jobsLabel=nameOfApp_20201228"
    }
    request_timeout => 60
    # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
    schedule => { cron => "0 * * * * UTC"}
    codec => "json"
    # A hash of request metadata info (timing, response headers, etc.) will be sent here
    metadata_target => "http_poller_metadata"
  }
}

As you can see in the urls part the endpoint follows the pattern:

 http://somehost/jobs?jobsLabel=sometext_yyyyMMdd

Where nameOfApp_20201228 is some label my script creates when it starts the job, and it contains the name of the app followed by the date of the day when the job has been created in this example it is 20201228.
Is there a way to write the config file of logstash that dynamically updates the jobsLabel to be today's date? or at least to watch all job-labels starts with nameOfApp_? something like:

urls => {
   test1 => "http://somehost/jobs?jobsLabel=nameOfApp_*"
}

or

urls => {
   test1 => "http://somehost/jobs?jobsLabel=nameOfApp_%{today}"
}

Note: I am runing Logstash and Elasticsearch with Docker

#2

May be you can set environment variable for that

urls => {
   test1 => "http://somehost/jobs?jobsLabel=nameOfApp_%${today_AppName}"
}

And find a way to upadte that env with a ruby filter, something like this may work

 ruby {
        code => '
            ENV["today_AppName"] = "nameOfApp_" + Time.now.strftime("%d/%m/%Y %H:%M");
        '
    }
#3

Thanks for your answer but I do not think I can set the environment variable in the filter since the filters are executed after the input plugins.

#4

I refered to this thread
Not sure if it's possible to update environment variable from ruby context

1 Like
#5

I think this issue has been requested by many developers since long time but still no solid solution for it other than some work arounds

#6

Yes, I'm one of them looking for that solid solution :slight_smile:
for the moment i generate dynamically the configuration that it get loaded everytime updated.
May be @Badger and @leandrojmp an advice on this topic

1 Like
#7

There is an open issue on GitHub for this problem since 2016

#8

I think that this is already the simplest solution for this case at the moment.

I would use a simple script to change the url line every day, which would trigger a config reload.

Or move the polling process to a external script and use the exec input plugin.

1 Like
#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.