# How to get RSS feed plugin from indexing duplicate posts

**URL:** <https://discuss.elastic.co/t/how-to-get-rss-feed-plugin-from-indexing-duplicate-posts/40776>\
**Category:** Logstash\
**Created:** [February 2, 2016, 6:30pm UTC](https://discuss.elastic.co/t/how-to-get-rss-feed-plugin-from-indexing-duplicate-posts/40776 "2016-02-02T18:30:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![MaxOrelus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxorelus/32/6441_2.png) [@MaxOrelus](https://discuss.elastic.co/u/MaxOrelus)\
**Post date:** [February 2, 2016, 6:30pm UTC](https://discuss.elastic.co/t/how-to-get-rss-feed-plugin-from-indexing-duplicate-posts/40776/1 "2016-02-02T18:30:45Z")

</div>

I'm currently playing with the [RSS plugin](https://www.elastic.co/guide/en/logstash/2.2/plugins-inputs-rss.html) for Logstash to index articles from my website. The plugin works good, but I notice if your interval is every 30 seconds or so, Logstash seems to re-index the same posts over and over again.

I guess my question is, how can I get Logstash to only index new articles that have been published to my rss feed rather than indexing the entire rss feed every time it runs through the interval time?

Thank you

---

<div class="post-metadata">

**Author:** ![yevmel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yevmel/32/9375_2.png) [@yevmel](https://discuss.elastic.co/u/yevmel)\
**Post date:** [April 24, 2016, 12:16pm UTC](https://discuss.elastic.co/t/how-to-get-rss-feed-plugin-from-indexing-duplicate-posts/40776/2 "2016-04-24T12:16:13Z")

</div>

Hi Max,

better late then never 🙂

i assume items in your feed contain changing fields. I had a similar issue with a feed, where the updated field changed frequently. You can either fix the feed if the changing field is a bug (and you are in control of the feed) or you can patch the rss plugin for logstash. Take a look [here](https://github.com/logstash-plugins/logstash-input-rss/blob/master/lib/logstash/inputs/rss.rb). Even without experience in ruby the plugin is pretty easy to understand. In my case, i just removed the "updated" field and duplicate entries were gone.

best regards,  
Yevgeniy

**UPDATE** :  
if this is your only problem with the rss plugin you probably do not need to fork/patch the plugin. you can use the mutate filter in your logstash configuration file to remove the changing field. in my case (changing field is "updated") the configuration looks like this:

```
filter {
    mutate {
        remove_field => ["updated"]
    }   
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:00am UTC](https://discuss.elastic.co/t/how-to-get-rss-feed-plugin-from-indexing-duplicate-posts/40776/3 "2017-07-06T05:00:59Z")

</div>