Dynamic input Http_poller url

[fernrguez]

Hi all,

I have a pipeline that executes every minute, at this moment I'm using the http_poller with a Get operation that, without filter, returns the first 100 in the last 24h.
It is possible to add filters on the Url in order to get the logs from certain time up to now, but I haven't seen any option of how generate an http_poller dynamic url to get the datetime minus an specific time.

Before I move the http_poller to a heartbeat and do the call as a filter, is there any other way to do it?

Thanks in advance

[Mohamad_Rahim]

I'm in the same exact position, till now I didn't find any positive sign that this is doable.
Badger suggested to use a generator input to trigger dummy events, then use an http filter to make the REST call, but it is very "workaroundy" for my taste.
Let me know if you find something!

[Mohamad_Rahim]

I think I found an even dirtier solution:
1-Set environment variables and set it to a default value, like: $date = "2019-01-01"
(https://www.elastic.co/guide/en/logstash/current/environment-variables.html)

2-In the input section of your Logstash config file, use the environment variable, like: url =>"https://jsonplaceholder.typicode.com/posts/from=${date}"

3-In the Filter section, use a the Ruby code plugin to update your $date environment variable (to something like $date = today()-10h )

4-Also in the Filter section, use the Ruby code plugin to edit your Logstash config file, by something that won't affect your pipeline, like adding or removing stdout in Output. Anything that triggers a config reload

5-Start Logstash with the --config.reload.automatic switch (https://www.elastic.co/guide/en/logstash/current/reloading-config.html)

6-TADA, you just got yourself the dirtiest workaround to ever walk this planet!

[Mohamad_Rahim]

Update:
The above contraption didn't work, unfortunately even with the --config.reload.automatic, the environment variables are not getting updated, because once the env vars gets updated it needs to be reloaded for the current process to see the update.
I tried adding the "refreshenv" command in the ruby code also, it does refresh, but I think to the cmd window only and not to the Java process, hence I'm still stuck in the same problem.
At this point, I'm super pissed and gave up on Logstash, I will use a PS or Python script to do this one.

[Badger]

@Mohamad_Rahim

The above contraption didn't work

It can be made to work. I configure a pipeline to read

path.config: "/home/user/test*.conf"

and create test.conf which has the following input

    http_poller { urls => { "first" => "http://${URL}" } schedule => { cron => "*/15 * * * * *" } }

On the command line I define URL to be 'localhost:9600/?pretty'. Then I add this ruby filter

    ruby {
        code => '
            ENV["URL"] = "localhost:9600/_node/os"
            File.open("test2.conf", "w") {}
        '
    }

When I run that configuration I get the output of the first URL followed 12 seconds later by the output of the second URL. So there is definitely potential here.