# \[ANN\] RSS River Plugin 0.0.6 **URL:** \ **Category:** Elasticsearch\ **Created:** [February 7, 2012, 10:15pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628 "2012-02-07T22:15:34Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2012, 10:15pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/1 "2012-02-07T22:15:34Z")
RSS River 0.0.6 has just been released [1]. RSS River will now works with the new Elasticsearch 0.19.0.RC1 release candidate. Please feel free to test it, fork it, create issues, contribute. 😉 [1] [http://dadoonet.github.com/rssriver/](http://dadoonet.github.com/rssriver/) [http://dadoonet.github.com/rssriver/](http://dadoonet.github.com/rssriver/) -- David Pilato [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : [https://twitter.com/#!/dadoonet](https://twitter.com/#!/dadoonet) [https://twitter.com/#!/dadoonet](https://twitter.com/#!/dadoonet) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 7, 2012, 11:34pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/2 "2012-02-07T23:34:16Z")
Cheers! On Wednesday, February 8, 2012 at 12:15 AM, David Pilato wrote: > RSS River 0.0.6 has just been released [1]. > > RSS River will now works with the new Elasticsearch 0.19.0.RC1 release candidate. > > Please feel free to test it, fork it, create issues, contribute… 😉 > > [1] [http://dadoonet.github.com/rssriver/](http://dadoonet.github.com/rssriver/) > > -- > > David Pilato > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > Twitter : [https://twitter.com/#!/dadoonet](https://twitter.com/#!/dadoonet) ---
**Author:** ![jeangld](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jeangld](https://discuss.elastic.co/u/jeangld)\ **Post date:** [February 8, 2012, 12:35am UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/3 "2012-02-08T00:35:43Z")
Hi, I'm new to Java, but experienced in .NET and PHP. I still have trouble getting the IDE and development environment set up. What I accomplished so far is: 1. Downloaded the intellij IDEA Community edition 2. Openend from existing source on git following projects: - elasticsearch - some elasticsearch-river-\* (twitter, wikipedia, couchdb) and - your rssriver. 1. I was able to build all sources and made run the tests in elasticsearch and rssriver. 2. By running tests I mean, full debug, step by step, variable inspection and so on What I'm missing is: - not able to test elasticsearch-river-twitter or -wikipedia in the IDE. (no test file?) - I don't know how to contribute changes back to the community ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 8, 2012, 2:51pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/4 "2012-02-08T14:51:02Z")
Hi, > What I'm missing is: > - not able to test elasticsearch-river-twitter or -wikipedia in the > IDE. (no test file?) There are no test in river twitter. No src/test/java folder in > **[GitHub - elastic/elasticsearch-river-twitter: Twitter River Plugin for...](https://github.com/elastic/elasticsearch-river-twitter/tree/master)** > > master [https://github.com/elasticsearch/elasticsearch-river-twitter/tree/master/src](https://github.com/elasticsearch/elasticsearch-river-twitter/tree/master/src) nor [GitHub - elastic/elasticsearch-river-wikipedia: Wikipedia River Plugin for elasticsearch (STOPPED)](https://github.com/elasticsearch/elasticsearch-river-wikipedia) [https://github.com/elasticsearch/elasticsearch-river-wikipedia](https://github.com/elasticsearch/elasticsearch-river-wikipedia) So if you want to test the twitter river, start an ES node, install the twitter plugin and start the river. If you want to "debug" the twitter river, build your own test classes and start them in debug mode. > - I don't know how to contribute changes back to the community > Not sure I understand this. I would answer : fork the project on github, > modify it, commit and push your changes and then make a pull request. Is > that what you are after ? HTH David. ---
**Author:** ![jeangld](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jeangld](https://discuss.elastic.co/u/jeangld)\ **Post date:** [February 8, 2012, 3:08pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/5 "2012-02-08T15:08:13Z")
Hi, thanks for the answer! > There are no test in river twitter. No src/test/java folder > So if you want to test the twitter river, start an ES node, install the > twitter plugin and start the river. I've done that, and it works like a charm 🙂 > If you want to "debug" the twitter river, build your own test classes and > start them in debug mode. There is my problem. Without experience in Java, I don't even can create a proper test class... > Not sure I understand this. I would answer : fork the project on github, > modify it, commit and push your changes and then make a pull request. Is > that what you are after ? Yes, I signed up on github and will play around a bit. I have to learn this too, I guess. The change what comes in my mind is some RSS "automation" work I've done already in PHP and will try to port it in Java. It is basically an automatic recognition when the feed changes, instead of using fixed intervals (hourly, daily, every minute, ...). It looks for changes, saves them in a database and makes calculations for the next run. So, very active feeds are pulled frequently and some others just once in a while. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 8, 2012, 8:25pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/6 "2012-02-08T20:25:27Z")
> > If you want to "debug" the twitter river, build your own test classes > > and > > start them in debug mode. > > There is my problem. Without experience in Java, I don't even can > > create a proper test class... > > Have a look on other projects : > > [https://github.com/dadoonet/rssriver/tree/master/src/test/java/org/elasticse](https://github.com/dadoonet/rssriver/tree/master/src/test/java/org/elasticse) > > arch/river/rss RomeTest.java is a real test class (I mean that's a JUnit test case so you need to launch it with JUnit). RssRiverTestLauncher.java is a standalone class (main()). You can launch it directly. It uses RssRiverTest.java and its parent class AbstractRssRiverTest.java. You can clone this code for your own needs. > > Not sure I understand this. I would answer : fork the project on > > github, > > modify it, commit and push your changes and then make a pull request. > > Is > > that what you are after ? > > Yes, I signed up on github and will play around a bit. I have to learn > > this too, I guess. > > Oh yes ! Learn Git and GitHub. Very powerful but it's IMHO a revolution in > > your mind. > > If you used to play with SVN or CVS, forget everything you have learned > > before and start to understand git concepts. But, ES mailing list is not the best place to talk about it. > The change what comes in my mind is some RSS "automation" work I've > done already in PHP and will try to port it in Java. It is basically > an automatic recognition when the feed changes, instead of using fixed > intervals (hourly, daily, every minute, ...). It looks for changes, > saves them in a database and makes calculations for the next run. So, > very active feeds are pulled frequently and some others just once in a > while. > So you mean that you try to fetch content every 5 minutes and if there is no > change, you change the period to 10 mn and so on ??? > I thought using the RSS specification as some feeds provides information on > rate change. See : > [RSS 2.0 Specification (Current)](http://www.rssboard.org/rss-specification#ltttlgtSubelementOfLtchannelgt) ---
**Author:** ![jeangld](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jeangld](https://discuss.elastic.co/u/jeangld)\ **Post date:** [February 8, 2012, 10:09pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/7 "2012-02-08T22:09:03Z")
Hi David, great, thank you so much again - I will look for some java/github tutorials, books and videos to get up to date. > So you mean that you try to fetch content every 5 minutes and if there is no > change, you change the period to 10 mn and so on ??? Thats what I used to do. Double the time (if no new posts) and then take half of time (if new posts). But thats not working with real bloggers. Nowadays I save the last seven days, with the corresponding hours. And then I decide when to fetch the feed again. On some blogs you see activity only during weekdays from 9am to 5pm in their timezone.Other autogenerated blogs may fill up the RSS feed every 5 minutes. It depends. I guess this will be way easier to implement here, since you already save the timestamp of the new link and Elasticsearch could give us some nice facetting over the last 7 days / 24 hours. Then we would only need one more field for the timestamp of the next fetch date. Its not that I save bandwith or computing time with such a strategy, but if we thiink of million of rss feeds it might matter. Jean ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 9, 2012, 3:05am UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/8 "2012-02-09T03:05:54Z")
Something we probably can do is to modify the update\_rate to be a cron expression. With a new property like auto\_adjust\_time:true/false we can implement what you described here. What do you think ? David 😉 @dadoonet Le 8 févr. 2012 à 23:09, "[jeangld@yahoo.com](mailto:jeangld@yahoo.com)" [jeangld@yahoo.com](mailto:jeangld@yahoo.com) a écrit : > Hi David, > > great, thank you so much again - I will look for some java/github > tutorials, books and videos to get up to date. > > > So you mean that you try to fetch content every 5 minutes and if there is no > > change, you change the period to 10 mn and so on ??? > > Thats what I used to do. Double the time (if no new posts) and then > take half of time (if new posts). But thats not working with real > bloggers. Nowadays I save the last seven days, with the corresponding > hours. And then I decide when to fetch the feed again. On some blogs > you see activity only during weekdays from 9am to 5pm in their > timezone.Other autogenerated blogs may fill up the RSS feed every 5 > minutes. It depends. I guess this will be way easier to implement > here, since you already save the timestamp of the new link and > Elasticsearch could give us some nice facetting over the last 7 days / > 24 hours. Then we would only need one more field for the timestamp of > the next fetch date. Its not that I save bandwith or computing time > with such a strategy, but if we thiink of million of rss feeds it > might matter. > > Jean ---
**Author:** ![jeangld](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jeangld](https://discuss.elastic.co/u/jeangld)\ **Post date:** [February 21, 2012, 2:57pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/9 "2012-02-21T14:57:37Z")
Hi David, sorry for taking me so long to reply, but I made my feet wet with java 🙂 I found a great java tutorial on sourceforge and was able to do a lot of things. I've choosen the free version of IntelliJ IDEA, but will also try out eclipse. In java I find myself just typing one or two letters and using the autocomplete feature a lot. I made also some speed tests with my old php scripts and java was on average 20x faster. Unbelievalble 🙂 It's almost C speed... I can't believe it, but I really managed to write tests for the twitter and wikipedia river plugins. I took your mentioned examples and I finally understood that in the tests we run new node, feed it with the river data and then wait until the code is triggered. > With a new property like > auto\_adjust\_time:true/false > we can implement what you described here. > What do you think ? Yes, something like this would be great. I will have a look again in my old php code, since a lot changes with the object oriented way to do things. Jean ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 22, 2012, 8:53pm UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/10 "2012-02-22T20:53:38Z")
Hi Jean, > > With a new property like > > auto\_adjust\_time:true/false > > we can implement what you described here. > > What do you think ? > > Yes, something like this would be great. I will have a look again in > > my old php code, since a lot changes with the object oriented way to > > do things. Ok. You can start by opening an issue here : > **[Build software better, together](https://github.com/dadoonet/rssriver/issues/new)** > > GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects. If you want to contribute, you're welcome ! 😉 David. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:38am UTC](https://discuss.elastic.co/t/ann-rss-river-plugin-0-0-6/6628/11 "2017-07-06T03:38:31Z")