# Logstash rss plugin not storing field

**URL:** <https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447>\
**Category:** Logstash\
**Created:** [November 5, 2017, 10:49pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447 "2017-11-05T22:49:26Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 5, 2017, 10:49pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/1 "2017-11-05T22:49:27Z")

</div>

I have successfully setup the logstash-rss-plugin and its working fine except one field is not being stored. I need it since the feed items are updated regularly, and without it I am getting duplicate entries in Elasticsearch. This is how the field is shown when I curl the feed from the command line.

```
<guid isPermaLink="false">A387658297364928.html</guid>

```

Can someone help me to add this as a field in elasticsearch? Im guessing it has something to do with my logstash config? BTW I have successfully grok'd the message field and setup the time fields.

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 6, 2017, 2:22am UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/2 "2017-11-06T02:22:30Z")

</div>

Found this two year old git issue, can I get some help 🙂

> <https://github.com/logstash-plugins/logstash-input-rss/issues/11>

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 7, 2017, 6:48pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/3 "2017-11-07T18:48:22Z")

</div>

bueller?

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 8, 2017, 9:00pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/4 "2017-11-08T21:00:24Z")

</div>

Surely there is someone who is able & willing to help!

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 10, 2017, 2:01pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/5 "2017-11-10T14:01:39Z")

</div>

Still holding out hope!

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 13, 2017, 6:41pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/6 "2017-11-13T18:41:27Z")

</div>

bump.

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 15, 2017, 1:22pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/7 "2017-11-15T13:22:26Z")

</div>

bump

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 17, 2017, 5:13pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/8 "2017-11-17T17:13:34Z")

</div>

bump again. Not going away!

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 19, 2017, 2:02am UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/9 "2017-11-19T02:02:09Z")

</div>

bump

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 20, 2017, 1:15pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/10 "2017-11-20T13:15:37Z")

</div>

bump

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 22, 2017, 7:43pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/11 "2017-11-22T19:43:59Z")

</div>

bump

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 24, 2017, 2:18pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/12 "2017-11-24T14:18:31Z")

</div>

bump

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [November 24, 2017, 4:59pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/13 "2017-11-24T16:59:10Z")

</div>

This is either simple or complex to fix depending on whether we want a more flexible solution i.e. add an `extensions` that allows for more than the guid to be added to an event.

With a bit of fiddling in Ruby's REPL `irb` I see this.

```ruby
feed.items[0].guid
=> #<RSS::Rss::Channel::Item::Guid:0x1338fb5 @parent=nil, @converter=nil, @content="\nhttp://liftoff.msfc.nasa.gov/2003/05/27.html#item571\n", @isPermaLink=nil, @do_validate=true
feed.items[0].guid.content.strip
=> "http://liftoff.msfc.nasa.gov/2003/05/27.html#item571"

```

Bummer, the `guid` is an object instance not a primitive value - i suppose because it is defined to have a `isPermaLink` attribute. The `source` and `enclosure` elements attributes too. So this is why its hard to add some sort of generic behaviour driven from the config.

In the rss input code we have:

```ruby
  def handle_rss_response(queue, item)
    @codec.decode(item.description) do |event|
      event.set("Feed", @url)
      event.set("published", item.pubDate)
      event.set("title", item.title)
      event.set("link", item.link)
      event.set("author", item.author)
      decorate(event)
      queue << event
    end
  end

```

If you want a quick fix and because its Ruby you can edit the rss input code in your Logstash installation.  
Find the file that contains the string `def handle_rss_response` under the `logstash/vendor` folder.  
Clue: its probably at logstash/vendor/bundle/jruby/?/gems/logstash-input-rss-?/lib/logstash/inputs/rss.rb  
Edit this file.  
Add this line before `decorate(event)`

```ruby
      event.set("guid", item.guid.content.strip) unless item.guid.nil?

```

**Note** If you update the plugin or Logstash you will lose the edit and need to redo it - unless the rss input plugin update contains the fix.

After the edit, the handle\_rss\_response method should look like this:

```ruby
  def handle_rss_response(queue, item)
    @codec.decode(item.description) do |event|
      event.set("Feed", @url)
      event.set("published", item.pubDate)
      event.set("title", item.title)
      event.set("link", item.link)
      event.set("author", item.author)
      event.set("guid", item.guid.content.strip) unless item.guid.nil?
      decorate(event)
      queue << event
    end

```

Restart Logstash and look for a load error on startup, no error -\> all good, error -\> paste error here.  
Hope this helps.

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 25, 2017, 5:00pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/14 "2017-11-25T17:00:52Z")

</div>

I added this to my config with no errors, I'll let you know if it worked tomorrow when the new index has been created. Thanks!

---

<div class="post-metadata">

**Author:** ![prophoto](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@prophoto](https://discuss.elastic.co/u/prophoto)\
**Post date:** [November 27, 2017, 2:02pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/15 "2017-11-27T14:02:23Z")

</div>

Working great, thanks!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2017, 2:02pm UTC](https://discuss.elastic.co/t/logstash-rss-plugin-not-storing-field/106447/16 "2017-12-25T14:02:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.