# River plugin: detect updates in the input data **URL:** \ **Category:** Elasticsearch\ **Created:** [June 13, 2013, 11:12am UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394 "2013-06-13T11:12:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Marc](https://avatars.discourse-cdn.com/v4/letter/m/8dc957/32.png) [@Marc](https://discuss.elastic.co/u/Marc)\ **Post date:** [June 13, 2013, 11:12am UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394/1 "2013-06-13T11:12:41Z")
Hi, is there any way to get notification that a river index has been modified? I have tried using the RSS River plugin and it seems that once the rss input has been defined, the plugin starts with that data set and any attempt to add/modify the input data has no effect (the new rss sources are stored but the RSS are not processed by the river which is not notified that the input data are changed). Is there any other way, other then restart the plugin or to delete\recreate the index, to process the updated items? Thanks in advance, Marc -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 13, 2013, 11:16am UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394/2 "2013-06-13T11:16:00Z")
Could you add an issue to the RSS project? I think I can support update. That said, you can remove the \_river itself and recreate it. It does not mean that you have to remove your index that contains data itself. ## Does it help? David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 13 juin 2013 à 13:12, Marc [ciarcia.marco@gmail.com](mailto:ciarcia.marco@gmail.com) a écrit : > Hi, > is there any way to get notification that a river index has been modified? > I have tried using the RSS River plugin and it seems that once the rss input has been defined, the plugin starts with that data set and any attempt to add/modify the input data has no effect (the new rss sources are stored but the RSS are not processed by the river which is not notified that the input data are changed). > Is there any other way, other then restart the plugin or to delete\recreate the index, to process the updated items? > > Thanks in advance, > Marc > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Marc](https://avatars.discourse-cdn.com/v4/letter/m/8dc957/32.png) [@Marc](https://discuss.elastic.co/u/Marc)\ **Post date:** [June 13, 2013, 11:52am UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394/3 "2013-06-13T11:52:03Z")
Thanks for the quick reply, I have added the issue on the rss project. BTW do you have any suggestion on how to be notified that the index containing the list of feed has been updated? By deleting and recreating the \_river the plugin kills and restarts all threads again, it would be nice to add\remove only threads with updated data Il giorno giovedì 13 giugno 2013 13:16:00 UTC+2, David Pilato ha scritto: > Could you add an issue to the RSS project? > I think I can support update. That said, you can remove the \_river itself > and recreate it. > It does not mean that you have to remove your index that contains data > itself. > > ## Does it help? > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 13 juin 2013 à 13:12, Marc \<[ciarci...@gmail.com](mailto:ciarci...@gmail.com) \\> a écrit > : > > Hi, > is there any way to get notification that a river index has been modified? > I have tried using the RSS River plugin and it seems that once the rss > input has been defined, the plugin starts with that data set and any > attempt to add/modify the input data has no effect (the new rss sources are > stored but the RSS are not processed by the river which is not notified > that the input data are changed). > Is there any other way, other then restart the plugin or to > delete\recreate the index, to process the updated items? > > Thanks in advance, > Marc > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 13, 2013, 1:38pm UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394/4 "2013-06-13T13:38:29Z")
Not sure I fully understood your question. Let me answer like this. You can create as many RSS river instance as you want. For example: $ curl -XPUT 'localhost:9200/\_river/lemonde/\_meta' -d '{ "type": "rss", "rss": { "feeds" : [ { "name": "lemonde", "url": "[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", "update\_rate": 900000, "ignore\_ttl": true } ] } }' And $ curl -XPUT 'localhost:9200/\_river/lefigaro/\_meta' -d '{ "type": "rss", "rss": { "feeds" : [ { "name": "lefigaro", "url": "[Le Figaro - RSS](http://rss.lefigaro.fr/lefigaro/laune)", "update\_rate": 900000, "ignore\_ttl": true } ] } }' Then, if you need to stop one specific river, you can do it as is: $ curl -XDELETE 'localhost:9200/\_river/lefigaro/\_meta' It should only remove the RSS river instance for that last URL. Is it what you are looking for? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 13 juin 2013 à 13:52, Marco Ciarcia [ciarcia.marco@gmail.com](mailto:ciarcia.marco@gmail.com) a écrit : > Thanks for the quick reply, I have added the issue on the rss project. > BTW do you have any suggestion on how to be notified that the index containing the list of feed has been updated? > By deleting and recreating the \_river the plugin kills and restarts all threads again, it would be nice to add\remove only threads with updated data > > Il giorno giovedì 13 giugno 2013 13:16:00 UTC+2, David Pilato ha scritto: > Could you add an issue to the RSS project? > I think I can support update. That said, you can remove the \_river itself and recreate it. > It does not mean that you have to remove your index that contains data itself. > > ## Does it help? > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > @dadoonet | @elasticsearchfr | @scrutmydocs > > Le 13 juin 2013 à 13:12, Marc [ciarci...@gmail.com](mailto:ciarci...@gmail.com) a écrit : > > > Hi, > > is there any way to get notification that a river index has been modified? > > I have tried using the RSS River plugin and it seems that once the rss input has been defined, the plugin starts with that data set and any attempt to add/modify the input data has no effect (the new rss sources are stored but the RSS are not processed by the river which is not notified that the input data are changed). > > Is there any other way, other then restart the plugin or to delete\recreate the index, to process the updated items? > > > > Thanks in advance, > > Marc > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Marc](https://avatars.discourse-cdn.com/v4/letter/m/8dc957/32.png) [@Marc](https://discuss.elastic.co/u/Marc)\ **Post date:** [June 13, 2013, 2:28pm UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394/5 "2013-06-13T14:28:02Z")
That will be creating separate indexes for articles (i.e. lefigaro and lemonde), making the result search complex. Normally, I'd like different rss feeds source, with articles collected within the same index and this could be done by adding an array of different "feeds" elements in the curl below, but if I update that list and send the curl again with different elements the request is ignored. $ curl -XPUT 'localhost:9200/\_river/lemonde/\_meta' -d '{ "type": "rss", "rss": { "feeds" : [{ "name": "lemonde", "url": "[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", "update\_rate": 900000, "ignore\_ttl": true } ] }}' $ curl -XPUT 'localhost:9200/\_river/articles/\_meta' -d '{ "type": "rss", "rss": { "feeds" : [{ "name": "lemonde", "url": "[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", "update\_rate": 900000, "ignore\_ttl": true }, { "name": "lefigaro", "url": "[Le Figaro - RSS](http://rss.lefigaro.fr/lefigaro/laune)" } ] }}' The second request is ignored, or better, a new version of the index ("articles" in the example above) will be created, but the river will not receive the update and will keep working as before till elasticsearch and the plugin doesn't restart or the \_river index gets deleted\re-created Il giorno giovedì 13 giugno 2013 15:38:29 UTC+2, David Pilato ha scritto: > Not sure I fully understood your question. Let me answer like this. > > You can create as many RSS river instance as you want. > For example: > > $ curl -XPUT 'localhost:9200/\_river/lemonde/\_meta' -d '{ "type": "rss", "rss": { "feeds" : [{ "name": "lemonde", "url": "[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", "update\_rate": 900000, "ignore\_ttl": true } ] }}' > > And > > $ curl -XPUT 'localhost:9200/\_river/lefigaro/\_meta' -d '{ "type": "rss", "rss": { "feeds" : [{ "name": "lefigaro", "url": "[Le Figaro - RSS](http://rss.lefigaro.fr/lefigaro/laune)", "update\_rate": 900000, "ignore\_ttl": true } ] }}' > > Then, if you need to stop one specific river, you can do it as is: > > $ curl -XDELETE 'localhost:9200/\_river/lefigaro/\_meta' > > It should only remove the RSS river instance for that last URL. > > Is it what you are looking for? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 13 juin 2013 à 13:52, Marco Ciarcia \<[ciarci...@gmail.com](mailto:ciarci...@gmail.com) \\> > a écrit : > > Thanks for the quick reply, I have added the issue on the rss project. > BTW do you have any suggestion on how to be notified that the index > containing the list of feed has been updated? > By deleting and recreating the \_river the plugin kills and restarts all > threads again, it would be nice to add\remove only threads with updated data > > Il giorno giovedì 13 giugno 2013 13:16:00 UTC+2, David Pilato ha scritto: > > > Could you add an issue to the RSS project? > > I think I can support update. That said, you can remove the \_river itself > > and recreate it. > > It does not mean that you have to remove your index that contains data > > itself. > > > > ## Does it help? > > > > _David Pilato_ | _Technical Advocate_ | \*[Elasticsearch.com](http://Elasticsearch.com)[http://elasticsearch.com/](http://elasticsearch.com/) > > \* > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > Le 13 juin 2013 à 13:12, Marc [ciarci...@gmail.com](mailto:ciarci...@gmail.com) a écrit : > > > > Hi, > > is there any way to get notification that a river index has been modified? > > I have tried using the RSS River plugin and it seems that once the rss > > input has been defined, the plugin starts with that data set and any > > attempt to add/modify the input data has no effect (the new rss sources are > > stored but the RSS are not processed by the river which is not notified > > that the input data are changed). > > Is there any other way, other then restart the plugin or to > > delete\recreate the index, to process the updated items? > > > > Thanks in advance, > > Marc > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 13, 2013, 2:34pm UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394/6 "2013-06-13T14:34:43Z")
Yeah unless you set index and type for that river. Sounds like I forget to document it! ☹ It's something like: { "type":"rss", "rss":{ "feeds":[ { "name":"lemonde", "url":"[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", "update\_rate":900000, "ignore\_ttl":true } ] }, "index":{ "index":"myindex", "type":"mytype" } } So basically, you should be able to define each feed to go to the same index / type. Does it help? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 13 juin 2013 à 16:28, Marco Ciarcia [ciarcia.marco@gmail.com](mailto:ciarcia.marco@gmail.com) a écrit : > That will be creating separate indexes for articles (i.e. lefigaro and lemonde), making the result search complex. > Normally, I'd like different rss feeds source, with articles collected within the same index and this could be done by adding an array of different "feeds" elements in the curl below, but if I update that list and send the curl again with different elements the request is ignored. > $ curl -XPUT 'localhost:9200/\_river/lemonde/\_meta' -d '{ > "type": "rss", > "rss": { > "feeds" : [ { > "name": "lemonde", > "url": "[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", > "update\_rate": 900000, > "ignore\_ttl": true > } > ] > } > }' > $ curl -XPUT 'localhost:9200/\_river/articles/\_meta' -d '{ > "type": "rss", > "rss": { > "feeds" : [ { > "name": "lemonde", > "url": "[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", > "update\_rate": 900000, > "ignore\_ttl": true > }, { > "name": "lefigaro", > "url": "[Le Figaro - RSS](http://rss.lefigaro.fr/lefigaro/laune)" > } > ] > } > }' > The second request is ignored, or better, a new version of the index ("articles" in the example above) will be created, but the river will not receive the update and will keep working as before till elasticsearch and the plugin doesn't restart or the \_river index gets deleted\re-created > > Il giorno giovedì 13 giugno 2013 15:38:29 UTC+2, David Pilato ha scritto: > Not sure I fully understood your question. Let me answer like this. > > You can create as many RSS river instance as you want. > For example: > $ curl -XPUT 'localhost:9200/\_river/lemonde/\_meta' -d '{ > "type": "rss", > "rss": { > "feeds" : [ { > "name": "lemonde", > "url": "[Le Monde.fr - Actualités et Infos en France et dans le monde](http://www.lemonde.fr/rss/une.xml)", > "update\_rate": 900000, > "ignore\_ttl": true > } > ] > } > }' > And > > $ curl -XPUT 'localhost:9200/\_river/lefigaro/\_meta' -d '{ > "type": "rss", > "rss": { > "feeds" : [ { > "name": "lefigaro", > "url": "[Le Figaro - RSS](http://rss.lefigaro.fr/lefigaro/laune)", > "update\_rate": 900000, > "ignore\_ttl": true > } > ] > } > }' > > Then, if you need to stop one specific river, you can do it as is: > $ curl -XDELETE 'localhost:9200/\_river/lefigaro/\_meta' > It should only remove the RSS river instance for that last URL. > > Is it what you are looking for? > > -- > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > @dadoonet | @elasticsearchfr | @scrutmydocs > > Le 13 juin 2013 à 13:52, Marco Ciarcia [ciarci...@gmail.com](mailto:ciarci...@gmail.com) a écrit : > > > Thanks for the quick reply, I have added the issue on the rss project. > > BTW do you have any suggestion on how to be notified that the index containing the list of feed has been updated? > > By deleting and recreating the \_river the plugin kills and restarts all threads again, it would be nice to add\remove only threads with updated data > > > > Il giorno giovedì 13 giugno 2013 13:16:00 UTC+2, David Pilato ha scritto: > > Could you add an issue to the RSS project? > > I think I can support update. That said, you can remove the \_river itself and recreate it. > > It does not mean that you have to remove your index that contains data itself. > > > > ## Does it help? > > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > > @dadoonet | @elasticsearchfr | @scrutmydocs > > > > Le 13 juin 2013 à 13:12, Marc [ciarci...@gmail.com](mailto:ciarci...@gmail.com) a écrit : > > > > > Hi, > > > is there any way to get notification that a river index has been modified? > > > I have tried using the RSS River plugin and it seems that once the rss input has been defined, the plugin starts with that data set and any attempt to add/modify the input data has no effect (the new rss sources are stored but the RSS are not processed by the river which is not notified that the input data are changed). > > > Is there any other way, other then restart the plugin or to delete\recreate the index, to process the updated items? > > > > > > Thanks in advance, > > > Marc > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:31am UTC](https://discuss.elastic.co/t/river-plugin-detect-updates-in-the-input-data/12394/7 "2017-07-06T02:31:26Z")