Response Headers http_poller

Elastic Stack Logstash
#1

Is it possible to access the response headers from events processed by the http_poller input plugin? I'm working with the Microsoft Table Service REST API:
https://docs.microsoft.com/en-us/rest/api/storageservices/fileservices/querying-tables-and-entities

The API returns the next row/partition in the response headers which point me where I need to go next for data.

Thank you

Http_poller and auth tokens
#2

Is it possible to access the response headers from events processed by the http_poller input plugin?

Yes, they're stored in the @metadata field by default but you can configure the input to store it elsewhere if you like.

However, if you want this information so that the next round of http_poller can request the next batch of data that won't be possible.

#3

Thank you.

So, it wouldn't be possible to add a field with an initial value, and then update it with a value in the response header?

#4

Having the http_poller input be aware of how to paginate is something we've discussed internally. I personally feel it's not something we can achieve because of the ways that things present pagination, such as:

  • Fetch the main page, get a token that you can query repeatedly in the future to get the next page of results (Elasticsearch Scroll API does this)
  • Fetch the first page, and the response includes an identifier for the next page (Github API does this, as does your Microsoft service, from your description)
  • Fetch pages by number (1, 2, 3, 4) and stop fetching when the current page has no results.

There are probably more pagination strategies, and I'm not sure we can prepare the http_poller plugin to support them all.

In cases such as this, I generally recommend that a new input plugin be created that has specific knowledge of how to read data from a specific data source. In this case, it would be my recommendation to have a custom input plugin that knows how to handle the pagination strategy deployed by the Microsoft Table Service.

#5

Thank you for your detailed response. I understand the challenges you bring up and they are good points. I haven't looked into creating custom plugins yet but I might see if I can whip something up to handle my specific use case.

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.