# Http\_poller only outputs hash? **URL:** \ **Category:** Logstash\ **Created:** [March 24, 2018, 6:17am UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426 "2018-03-24T06:17:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [March 24, 2018, 6:17am UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426/1 "2018-03-24T06:17:44Z")
Below is my pipeline. Trying to pull a list from a site and then parse it out with xml filter. ``` input { http_poller { urls => { isctop100 => "https://isc.sans.edu/api/topips/records/100?xml" } schedule => {"every" => "1s"} target => "data" codec => multiline { pattern => "" what => "next" } } } filter { xml { source => "data" store_xml => "false" xpath => [ "ipaddress/rank/text()", "Rank", "ipaddress/source/text()", "IPAddress", "ipaddress/reports/text()", "Reports", "ipaddress/targets/text()", "Targets" ] } if "?xml version" or "topips" in [data] { drop { } } } ``` I get the following error in Logstash logs: `XML filter expects a string but received a Hash` I see that the XML filter is getting a hash fed to it and I suspect it's the http\_poller input, anybody have a definitive answer? ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [March 24, 2018, 8:07am UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426/2 "2018-03-24T08:07:02Z")
The `target` directive for `http_poller` is a little... _weird_; I would avoid it, and rely on the codec putting the message in the `message` field; if you need it to end up in `data`, you can rename it immediately after the input in a filter: ```auto input { http_poller{ # ... (no target directive) } } filter { mutate { rename => { "message" => "data" } } } filter { xml { source => "data" # ... } } ``` * * * Without the `target` directive, an `Event` is created by the codec; most simple codecs (like multiline) will capture the message and create an `Event` that looks something like: ```auto { "message" => "the parsed message", "@timestamp" => Timestamp.current, "@metadata` => { # ... } } ``` When the `target` is set, the codec still creates the above `Event`, but then `http_poller` converts the `Event` to a `Hash` (which throws out the `@metadata`), and puts the _result_ in a _new_ event at the target address: ```auto { "@timestamp" => Timestamp.current, # won't necessarily match the inner timestamp "data" => { "message" => "the parsed message", "@timestamp" => Timestamp.current }, "@metadata` => { # ... _new_ metadata } } ``` ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [March 25, 2018, 10:03pm UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426/3 "2018-03-25T22:03:30Z")
Good stuff yaauie, thanks for the info. I BELIEVE I started without the target option and was encountering the same issue. Unfortunately, after requesting information from the link provider, they took the feed down for maintenance, I guess I notified them of an issue they weren't aware of. So right now, I have nothing to test against or the time to go out and find another one, I might be able to later tonight though. Another question, if the page is only updated say, once a day, but I poll the page every hour, does that mean I will have duplicate entries or does http\_poller (or some other input/codec/filter) have the ability to track and process only changed data? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 26, 2018, 7:52pm UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426/4 "2018-03-26T19:52:09Z")
> Another question, if the page is only updated say, once a day, but I poll the page every hour, does that mean I will have duplicate entries or does http\_poller (or some other input/codec/filter) have the ability to track and process only changed data? No, but if you save the data in an ES document with a fixed name you'll overwrite the same document again and again. ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [March 27, 2018, 2:20am UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426/5 "2018-03-27T02:20:31Z")
Interesting, leaves me with two questions. 1. How do you save the document into ES with a fixed name? 2. Guess it depends on how you accomplish number 1, but how could you configure it to create a new document at a given interval? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 27, 2018, 6:39am UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426/6 "2018-03-27T06:39:16Z")
Use the elasticsearch output's `document_id` option. Not sure I understand why you want to create a new document at fixed intervals (regardless of whether the source has changed), but you could do e.g. ``` document_id => "someprefix-%{+YYYYMMdd}" ``` to create a new document once a day. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 24, 2018, 6:40am UTC](https://discuss.elastic.co/t/http-poller-only-outputs-hash/125426/7 "2018-04-24T06:40:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.