How to set a schedule on Filter or reuse the content of a event

Elastic Stack Logstash
1

Hello everyone, need a share of your help again. This time with Logstash.

The context of the problem is:

I got a pipeline which goes into an API to retrieve a token with a POST, once I got this token I use it on another request (this time a GET) which will return a list of devices. So far I could make it happen by using the plugin HTTP POLLER on input and HTTP on Filter. The problem here is that this token have a duration of 3hours so theres no need to generate one that quickly, but theres a need to have the list of devices as soon as possible, such as 1minute at max.

Here is my config file:

input {
    http_poller{
        urls => {
           app => {
            method => post
            url => "[url to get the token]"
            headers => {
                
                "Content-Type" => "application/json"
                }
            body => '{ "client": "xxxxx","secret": "xxxxxx","tenant": xxxxx }'
           }
            
         }
        cacert => "cert/certificate.crt"
        request_timeout => 60
        schedule => {every => "1m"}
        codec => "json"
    }
    
}
filter{
  http {
    body_format => "json"
    url => "url to get the devices"
    verb => "GET"
    headers => { 
        "Authorization" => "Bearer %{access_token}"
        }
    cacert => "cert/certificate.crt"
    ecs_compatibility => disabled
  }

    split{
        field => "body"
    }
}

The dilemma is: I need the token generated to apply on the GET request, but I can only reuse it with the "%" on the header of the next request because is on the filter. If I could make that GET in another input I would be able to set a different schedule, but by doing it I can't access the token because it refers to another event.

The possible solutions I thought of were:

1 - Find a way to set a individual schedule for the filter to act (which I'm pretty sure it doesn't exist).
2 - Find a way to get that token generated in the first request into a kind of variable? So I can use it on another input with a individual schedule.

Open to new ideas, will be glad if anyone can help me pull this out.

2

It could probably be done....

Run an http_poller on a long schedule (3 hours). Run a heartbeat input on a short schedule (once a minute). Use a ruby filter with logic like this to copy the token to heartbeat events.

3

Thank you for the fast reply.

Would you have some example of the heartbeat input being used as you say? I checked arround how I could use it but couldn't find hints execpt for the schedule.

Looking arround the link you sent was really useful. I belive that the token would come out with something like this?

ruby { code => "@token = event.get('access_token')" }

That being said, if that's right the filter session maybe could be like:

filter{
  if [type] == "heartbeat"{  # To ensure the input of the http_poller wont trigger it.
  http {
    body_format => "json"
    url => "url to get the devices"
    verb => "GET"
    headers => { 
        "Authorization" => "Bearer @token?(not sure how to fit it here)"
        }
    cacert => "cert/certificate.crt"
    ecs_compatibility => disabled
  }

    split{
        field => "body"
    }
}
}

Please correct me if something is terrible wrong on this :sweat_smile:

4

I was thinking of something like

input {
    http_poller {
        ...
    }
    heartbeat {
        interval => 60
        tags => [ "heartbeat" ]
    }
}
filter {
    if "heartbeat" in [tags] {
        ruby { code => 'event.set("tokenField", @@saveToken)" }
    } else {
        ruby { code => '@@saveToken = event.get("tokenField")' }
    }
    if [tokenField] {
        http {
            ...
        }
    }
}
5

Thanks a lot, that logic worked just like magic!