# Http\_poller input only new lines? **URL:** \ **Category:** Logstash\ **Created:** [January 11, 2018, 10:36am UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058 "2018-01-11T10:36:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![eebee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eebee/32/26799_2.png) [@eebee](https://discuss.elastic.co/u/eebee)\ **Post date:** [January 11, 2018, 10:36am UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/1 "2018-01-11T10:36:48Z")
I'm using the http\_poller Logstash input plugin to ingest a logfile into Elasticsearch. But every time it polls data from the logfile, it polls the whole file. Config file: ``` input { http_poller { urls => { test => { method => get url => "http://xxx.xxx.xx/api/log" headers => { "Accept" => "application/json" "x-xx-api" => "xxxxx" } } } request_timeout => 20 # Supports "cron", "every", "at" and "in" schedules by rufus scheduler schedule => { cron => "* * * * * UTC"} codec => "json_lines" # A hash of request metadata info (timing, response headers, etc.) will be sent here metadata_target => "http_poller_metadata" } } output { elasticsearch { hosts => ["192.168.1.174:9200"] index => "xx-testing-%{+YYYY.MM}" } stdout { codec => rubydebug } } ``` Log file looks like this: ``` {"@message":"Successful api request","@timestamp":"2018-01-11T10:11:00.260Z","@fields":{"origin":"xx.xx.xx.xx","environment":"production_beta","label":"askquestiongui","level":"info"}} {"@message":"Successful api request","@timestamp":"2018-01-11T10:12:00.317Z","@fields":{"origin":"xx.xx.xx.xx","environment":"production_beta","label":"askquestiongui","level":"info"}} ``` If I use codec "json", I only get the first log-line once, codec "json\_lines" writes the complete logfile to Elasticsearch each time. PLease advice. 🙂 ---
**Author:** ![s1m0ne](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1m0ne/32/26471_2.png) [@s1m0ne](https://discuss.elastic.co/u/s1m0ne)\ **Post date:** [January 11, 2018, 10:41am UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/2 "2018-01-11T10:41:34Z")
Hi, why do not use FileBeat to produce events based on your log and send directly to Elastic? Cheers, s1m0ne ---
**Author:** ![eebee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eebee/32/26799_2.png) [@eebee](https://discuss.elastic.co/u/eebee)\ **Post date:** [January 11, 2018, 11:46am UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/3 "2018-01-11T11:46:31Z")
Doesn't filebeat have to be run at the server where the log file is? I cannot install anything on the server where the log-file is located. ---
**Author:** ![eebee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eebee/32/26799_2.png) [@eebee](https://discuss.elastic.co/u/eebee)\ **Post date:** [January 11, 2018, 2:45pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/4 "2018-01-11T14:45:08Z")
Another issue for me is that I have to pull the log file from the server, which is on a public server, to my Elasticsearch server that is on a private network. I cannot use filebeat to push the data tom Elasticsearch. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 11, 2018, 2:59pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/5 "2018-01-11T14:59:40Z")
Your http server will need to stateful. Say you use a query string of `http://xxx.xxx.xx/api/log?lines=100` and the server, having remembered that in the previous call it served lines 0 to 99, serves lines 100 to 199 to this call. The http\_poller input is not stateful and has no facility to remember what the last processed line number was and adjust the query string for example. ---
**Author:** ![eebee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eebee/32/26799_2.png) [@eebee](https://discuss.elastic.co/u/eebee)\ **Post date:** [January 11, 2018, 3:05pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/6 "2018-01-11T15:05:20Z")
I think I got it right now. I added: ``` document_id => "%{@timestamp}" ``` ..to my elasticsearch output. Then Elasticsearch doesn't duplicate that document\_id. Before it gave every new reading a unique document\_id. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 11, 2018, 3:05pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/7 "2018-01-11T15:05:56Z")
Consider a different architecture. Put Filebeat in the public zone on the server, Logstash in the DMZ and ES in the private zone. ---
**Author:** ![eebee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eebee/32/26799_2.png) [@eebee](https://discuss.elastic.co/u/eebee)\ **Post date:** [January 12, 2018, 7:42am UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/8 "2018-01-12T07:42:52Z")
We are aiming at having Elastic on a public server, but I cannot use Filebeat since we cannot install anything in the environment where the logfile are located. But our logging system uses the Winston library which can send logging messages directly to Logstash, so whenever I get a public server running, I think that may be an excellent way to go. 🙂 ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 12, 2018, 10:31am UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/9 "2018-01-12T10:31:33Z")
I just read the Winston docs and some of the code. It looks like it will try to dispatch the log line string to a destination immediately. The HTTP transport is acting as a client not a server AFAICT. I don't see how you are achieving persistence - via a Winston File transport? If so then the file is a persistent buffer. Then, with what tech does the LS http\_poller connect to so it retrieves the log lines from those files? I ask these questions not out of malice or because I doubt your solution but because I and others here can get to appreciate an alternative method to ship log lines from the edge. Regarding Elasticsearch clusters in the public zone, if you have not already done it, you must [secure it](https://www.elastic.co/guide/en/x-pack/6.1/xpack-security.html) Regarding your future plans. > But our logging system uses the Winston library which can send logging messages directly to Logstash By this I think you mean Winston HTTP transport (client mode) to LS http input (server mode). If so, there is a problem with buffering. LS will have to be up 24/7. How does the Winston client transport behave when the HTTP server is not available? Consider a load balancer between Winston and 2/3 LS instances (haproxy or nginx). If you consider a load balancer, then remember that consecutive log lines will be sent to different LS instance - no ordering. ---
**Author:** ![eebee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eebee/32/26799_2.png) [@eebee](https://discuss.elastic.co/u/eebee)\ **Post date:** [January 12, 2018, 11:25am UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/10 "2018-01-12T11:25:37Z")
Thanks for the input. 🙂 We have a few limitations in this project that currently cause us some problems, but I think this method I'm using is a fairly good way to overcome thoose issues. Having Logstash use the http\_poller input with a private api-key to fetch the data. I don't see much difference (performance wise) between having logstash to pull data from a server compared to having the logg-server to send the data to the Elastic server. Right now this setup will only run for a few weeks as a proof of concept. If we launch it properly, we have to scale everything alot anyway. 😉 ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [January 12, 2018, 3:00pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/11 "2018-01-12T15:00:02Z")
Great. However you did not answer the question of what tech you are using the serve the requests from the http\_poller? ---
**Author:** ![eebee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eebee/32/26799_2.png) [@eebee](https://discuss.elastic.co/u/eebee)\ **Post date:** [January 12, 2018, 4:14pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/12 "2018-01-12T16:14:38Z")
We are running an application in IBM Cloud, and it's a little restricted what you can and cannot do there. 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 9, 2018, 4:14pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058/13 "2018-02-09T16:14:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.