# REST API, http\_poller and ruby code
**URL:** https://discuss.elastic.co/t/rest-api-http-poller-and-ruby-code/176447
**Category:** Logstash
**Created:** [April 11, 2019, 3:54pm UTC](https://discuss.elastic.co/t/rest-api-http-poller-and-ruby-code/176447 "2019-04-11T15:54:15Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [April 11, 2019, 3:54pm UTC](https://discuss.elastic.co/t/rest-api-http-poller-and-ruby-code/176447/1 "2019-04-11T15:54:15Z")
I wonder if the following functionality, which I've successfully implemented in the form of a **client-side Python script** , can be completely replaced by **server-side Logstash configuration** , using a combination of [http\_poller input](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html) and [ruby filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html) plugins.
This is what my Python script currently does:
- Pulls a REST API every x seconds and asks for the latest record
- If the record is more than 1 steps ahead, it runs a for-loop and downloads one-by-one all the incremental new records, ensuring that no record is missing
- It stores the `last_run`, i.e. the last record downloaded so that if the script fails, it can start from that point
- It can historically load records if I provide it with a starting record and number of records that I want it to download
- It creates a log which I then use `Filebeat` to ship to elastic
The reasons why I used client-side scripting and not server-side Logstash plugins, are the following:
- the `http_poller input plugin` seems to be stateless (see this [post](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/5?u=nandrik) from @guyboertje) but I wonder if by using `ruby` to write to a file you can get around that
- there's no easy way to do a first query to ask for the latest record id and then run a loop to retrieve the new records, avoiding the sleep\_time. Again maybe in `ruby`, all this is possible.
On the flip side, there are some advantages if I could do this server-side in`Logstash`, namely:
- A server-side script can be easily used across multiple sources without need for installing and monitoring the script and without client dependencies such as rotating logs
- This can become the basis of a custom plug-in, further enriching and enhancing the received data in a cloud-based way.
Wonder if anyone has experiences with this dilemma and whether you've also resorted to client-side scripting or think this is achievable through some advanced understanding and implementation using Logstash functionality and its plugins.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 11, 2019, 4:06pm UTC](https://discuss.elastic.co/t/rest-api-http-poller-and-ruby-code/176447/2 "2019-04-11T16:06:55Z")
You cannot do it using an http\_poller input. No ruby filters will have executed when the input runs and there is no way to pass state to it.
You might be able to use an http filter. You could use any of the inputs that have a schedule option to create dummy events, then use a combination of ruby and http filters to do the work on that schedule.
It feels a bit like like [solving](http://www.jhnc.org/hanoi/sendmail-cf.html) the Towers of Hanoi problem in sendmail.cf. It can be done, and it is interesting to see it work, but that doesn't make it a good idea 😃
---
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [April 11, 2019, 4:18pm UTC](https://discuss.elastic.co/t/rest-api-http-poller-and-ruby-code/176447/3 "2019-04-11T16:18:27Z")
@Badger, thanks for the reply, I was not aware of the similar functionality of the [HTTP Filter Plugin](https://www.elastic.co/guide/en/logstash/master/plugins-filters-http.html) and it does make sense that this can be more easily combined with **Ruby** filters.
I guess I **need to find some examples** that use these filters in a combined way to understand and assess the tradeoff in complexity of this approach since my so far experience with `ruby` inside Logstash has been that it's quite hard to debug and ensure that it's working properly.
And I wonder if at that point you reach the stage of contemplating the development of a **custom Logstash plugin**.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 9, 2019, 4:18pm UTC](https://discuss.elastic.co/t/rest-api-http-poller-and-ruby-code/176447/4 "2019-05-09T16:18:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.