Two Http_poller inputs with access token

Elastic Stack Logstash
1

Hello!

I am trying to get data from an api that needs to be accessed using a token that we get from another api.

How can I pass this access token to get the data from the other api?

I had thought of doing something similar to this but I don't know if it's entirely correct.

input {
  http_poller {
    urls => {
      token => "https://url-token.com/api/token"
    }
    add_field => { "token" => "%{message}" }
    type => "Token-API"
  }

  http_poller {
    urls => {      
      url-api-data => {
        method => get
        cacert => "path/cert.crt"
        user => "x"
        password => "y"
        url => "https:/url.com/api"
        headers => {
          "Content-Type" => "application/json"
          "Authorization" => "Bearer %{token}"
        }
      }
    }
    request_timeout => 60
    schedule => { every => "5s"}
    codec => "json"
    metadata_target => "http_poller_metadata"
    type => "Data-API"
  }
    

}

Is there any better way to do it?
I don't know if in this case there could be data duplication/loss or if it will actually catch the token correctly.

Thanks in advance.

2

You cannot do that using http_poller. See this for an example and this for more discussion.

3

I don't quite understand why it doesn't work. Is it because the input would not return the token correctly or because the http_poller does not support tokens?

If I change the first input to get the token to an 'exec' input that curls the url and stores this token in a field and uses it in the poller header, would it work?

or should i keep the token at the input with hhtp_poller and then with http filter connect to the other api to keep the data?

I'm not very good with ruby ​​and I don't quite understand the example you gave me

4

The http_poller input is initialized before any events are created, so it cannot reference fields from them.

5

I'm not understanding it well, I'm sorry for my level of English.

I understand that with the input http_poller, I get the token and with http filter I get the data of interest from the api?

for example:

input {
  http_poller {
    urls => {
      token => "https://url-toke.com/api/token"
    }
    type => "Token-API"
  }
}

filter {

  http {
    url => "api/with/data"
    verb => "GET"
    target_body => "field"
    headers {
      "Authorization" => "Bearer %{message}"
    }
  }

}

output{
  elasticsearch{}
}

Could you send me a link with some complete example of http filter?

thank you!

6

I do not have one.

The schedule option on the http_poller option is required.

You may be able to use the http_poller to get the token.

The http filter will only get called when an event goes through the pipeline. That will happen each time the http_poller is triggered. If you do not want to call the http_poller input once for every call to the http filter then you must store the token, using code similar to what I shared a link to.

7

i think that the token will be a random one everytime i conect to the api.

I think I'm starting to understand...

I understand that if I want to obtain the data I must perform the http_poller to obtain the token, once the token is obtained it is stored in the indicated field ( http_poller_token_metadata) , if there is event the http filter is activated and It store the data in the indicated field ( filed-with-data) that will later be indexed in elastic .

Is that so? the data is actually fetched by the http filter? I did not understand well that the filter function is like an input

input {
  http_poller {
    urls => {
      token-url=> {
        method => get
        cacert => "/path/x.cer"
        user => "user"
        password => "pass"
        url => "http://api-url.com/token"
     }
    }
    request_timeout => 60
    schedule => { every => "5s"}
    metadata_target => "http_poller_token_metadata"
  }
}

filter {

  http {
    url => "api/with/data"
    verb => "GET"
    target_body => "filed-with-data"
    headers {
      "Authorization" => "Bearer %{http_poller_token_metadata}"
    }
  }

}

output{
  elasticsearch{}
}
8

That is correct.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.