Pagination on logstash http filter

Elastic Stack Logstash
1

I am generating a token by using http_poller input plugin, using the generated token in http filter part and splitting the fields from results, Can anyone please suggest me on Pagination.

As the api results are around 4000,

Blockquote

Preformatted text{
"total": 4425,
"results": [
{
"_id": {
"$oid": "65aac96f90ebf09ba386170e"
},................................................ },................................................


`> Blockquote`

type or paste code here
````Preformatted text`


my logstash pipeline:

input {
  http_poller {
    urls => {
      generatetoken => {
        method => post
        body => '{ "userID" :"${USER}", "userPWD" : "${PWD}" }'
        headers => {
          "Content-Type" => "application/json"
        }
        url => "https://**********/auth/login"
        request_timeout => 60
      }
    }
    truststore => "/usr/share/logstash/config/Kafka/cacerts"
    truststore_password => "changeit"
    schedule => { cron => "* * * * * UTC"}
    metadata_target => "my-token"
  }
}
filter{
     http {
       body_format => "json"
       follow_redirects => false
       verb => "POST"
       headers => { "Content-Type" => "application/json"
                   "Authorization" => "%{token}" }
       body => { "appCode" => "*****"}
       url => "https://*****"
       truststore => "/usr/share/logstash/config/Kafka/cacerts"
       truststore_password => "changeit"
       target_body => "[api_response]"
       # target_headers => "[@metadata][token]"
     }
     split { field => "[api_response][results]"  }
}
elasticsearch {
      hosts => ["${ES_HOST}"]
      user => "${USER}"
      password => "${PWD}"
      ilm_enabled => true
      ilm_policy => "*****_lifecycle_policy"
      ilm_rollover_alias => "${INDEX_ENV}-jobhistory-api"
      document_id => "%{[api_response][results][_id][$oid]}"
      ssl => true
      ssl_certificate_verification => true
      cacert => "/usr/share/logstash/config/SSL/${SSL_ENV}/elastic-cert.cer"
  }
}`Preformatted text`
2

The http_poller input plugin does not support pagination, you can't do that with Logstash.

An alternative would be using Filebeat in front of Logstash making the requests since the httpjson input of filebeat supports pagination.

1 Like
3

Thanks [Leandro Pereira] for your reply.

I wanted to achieve this by logstash, i sthere any other way around using logstash?

4

As mentioned no, the http_poller filter does not support pagination.

1 Like
5

I was referring to the below link

6

Hello,

Not sure what you are referring to, the link you shared is unrelated to the http_poller input.

As mentioned the http_poller input does not support pagination, there is nothing native in logstash that supports pagination.

You may be able to kind of replicate a pagination combining some logic and filters, as the link you mentioned, but I don't think this is the best approach.

Logstash is not the best tool for this, it would be easier to use Filebeat in this case to make the polling or write a quick python script that would make the request and save into a file for Logstash to process.

1 Like
7

Thanks a lot for your quick response,

Filebeat doesn't work for me as I am fetching the results by an api.

Do you have an example for phython script which saves a file for logstash to process.

Appreciate your help.

8

Did you check the documentation for the httpjson input that was shared before? This input does exactly that, it can make requests to APIs.

I do not, but you can easily find some searching on Google, this is unrelated to Logstash, you just need to make a request to your API and then save it on a file.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.