ElasticSearch, RSS and `_mapping` requirement?

Nev_Ada · July 28, 2013, 6:47am

I want to store RSS feeds in a searchable ElasticSearch store.

Using RSSRiver http://www.pilato.fr/rssriver, I am confused as to what
they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one http://venturebeat.com/feed, is
the approach to write a script to generate /_mapping or should it just work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "http://venturebeat.com/feed/"}]}})

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · July 28, 2013, 7:34am

Documentation is more complete here: https://github.com/dadoonet/rssriver
You need to create a _meta document in a specific index _river.

$ curl -XPUT 'localhost:9200/_river/lemonde/_meta' -d '{ "type": "rss", "rss": { "feeds" : [ { "name": "lemonde", "url": "http://www.lemonde.fr/rss/une.xml" } ] } }'
Don't forget _river index name.

HTH

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 juil. 2013 à 08:47, Nev Ada nevada.over9000@gmail.com a écrit :

I want to store RSS feeds in a searchable ElasticSearch store.

Using RSSRiver, I am confused as to what they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one, is the approach to write a script to generate /_mapping or should it just work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "http://venturebeat.com/feed/"}]}})

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Nev_Ada · July 28, 2013, 8:57am

Thanks

On Sunday, July 28, 2013 5:34:05 PM UTC+10, David Pilato wrote:

Documentation is more complete here: GitHub - dadoonet/rssriver: Elasticsearch Rss River (PROJECT STOPPED)
You need to create a _meta document in a specific index _river.

$ curl -XPUT 'localhost:9200/_river/lemonde/_meta' -d '{ "type": "rss", "rss": { "feeds" : [ { "name": "lemonde", "url": "Le Monde.fr - Actualités et Infos en France et dans le monde" } ] }}'

Don't forget _river index name.

HTH

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 juil. 2013 à 08:47, Nev Ada <nevada....@gmail.com <javascript:>> a
écrit :

I want to store RSS feeds in a searchable Elasticsearch store.

Using RSSRiver http://www.pilato.fr/rssriver, I am confused as to what
they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one http://venturebeat.com/feed,
is the approach to write a script to generate /_mapping or should it just
work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "VentureBeat"}]}})

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kumarsuryamani · July 29, 2013, 4:43am

I have uploaded a video for RSS river mapping, indexing and searching
example; you can watch it here http://www.youtube.com/watch?v=JaX70FpEkAA
It is only visual no audio.

thanks & regards,
surya

On Sunday, July 28, 2013 12:17:15 PM UTC+5:30, Nev Ada wrote:

I want to store RSS feeds in a searchable Elasticsearch store.

Using RSSRiver http://www.pilato.fr/rssriver, I am confused as to what
they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one http://venturebeat.com/feed,
is the approach to write a script to generate /_mapping or should it just
work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "VentureBeat"}]}})

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · July 29, 2013, 6:17am

We could add your video to RSS River page:

github.com

dadoonet/rssriver/blob/gh-pages/index.html

---
title: Open Source, Distributed, RESTful, Search Engine
layout: default
title_in_header: false
---
<div id="home">
    <div class="img_right">
        <img src="/rssriver/images/logo-icon.png" height="120px" alt="" />
        <div class="text">
            <h2>
                Elasticsearch
            </h2>
            <p>
               <b>Elasticsearch</b> is certainly the best search engine that cover all your needs
               and many more ! RESTful, JSon standard, Distributed, Zero Conf and fast, blazing fast...
            </p>
            <p>
                Project <a href="http://www.elasticsearch.org">documentation</a> is here and 
                source code is on <a href="https://github.com/elasticsearch/elasticsearch">github</a>.
            </p>

This file has been truncated. show original

Or

github.com

dadoonet/rssriver/blob/master/README.md

RSS River for Elasticsearch (PROJECT STOPPED)
===========================

Welcome to the RSS River Plugin for [Elasticsearch](http://www.elasticsearch.org/)

In order to install the plugin, run: 

```sh
bin/plugin -install fr.pilato.elasticsearch.river/rssriver/1.3.0
```

You need to install a version matching your Elasticsearch version:

|       Elasticsearch    |  RSS River Plugin |                                                            Docs                                                              |
|------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------|
|    master              | Build from source | See below                                                                                                                    |
|    es-1.x              | Build from source | [1.5.0-SNAPSHOT](https://github.com/dadoonet/rssriver/tree/es-1.x/#version-150-snapshot-for-elasticsearch-1x)                |
|    es-1.4              | Build from source | [1.4.0-SNAPSHOT](https://github.com/dadoonet/rssriver/tree/es-1.x/#version-140-snapshot-for-elasticsearch-14)                |
|    es-1.3              |     1.3.0         | [1.3.0](https://github.com/dadoonet/rssriver/tree/v1.3.0/#version-130-for-elasticsearch-13)                  |
|    es-1.1              |     1.1.0         | [1.1.0](https://github.com/dadoonet/rssriver/tree/v1.2.0/#rss-river-for-elasticsearch)                                       |

This file has been truncated. show original

What do you think? Open a pull request?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 06:43, Kumar Surya Mani ksuryamani@gmail.com a écrit :

I have uploaded a video for RSS river mapping, indexing and searching example; you can watch it here http://www.youtube.com/watch?v=JaX70FpEkAA
It is only visual no audio.

thanks & regards,
surya

On Sunday, July 28, 2013 12:17:15 PM UTC+5:30, Nev Ada wrote:

I want to store RSS feeds in a searchable Elasticsearch store.

Using RSSRiver, I am confused as to what they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one, is the approach to write a script to generate /_mapping or should it just work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "VentureBeat"}]}})
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kumarsuryamani · July 29, 2013, 6:32am

Hi David,

It sounds interesting; I would love it to be added to RSS River however I
don't know whether you had a chance to look into that.

thanks & regards,
surya

On Mon, Jul 29, 2013 at 11:47 AM, David Pilato david@pilato.fr wrote:

We could add your video to RSS River page:
https://github.com/dadoonet/rssriver/blob/gh-pages/index.html
Or
https://github.com/dadoonet/rssriver/blob/master/README.md

What do you think? Open a pull request?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 06:43, Kumar Surya Mani ksuryamani@gmail.com a
écrit :

I have uploaded a video for RSS river mapping, indexing and searching
example; you can watch it here
http://www.youtube.com/watch?v=JaX70FpEkAA
It is only visual no audio.

thanks & regards,
surya

On Sunday, July 28, 2013 12:17:15 PM UTC+5:30, Nev Ada wrote:

I want to store RSS feeds in a searchable Elasticsearch store.

Using RSSRiver http://www.pilato.fr/rssriver, I am confused as to what
they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one http://venturebeat.com/feed,
is the approach to write a script to generate /_mapping or should it
just work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "VentureBeat"**}]}})

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iBAjQ-jvbMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Warms n Regards,
Kumar Surya Mani

  **They conquer who believe they can. - John Dryden

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · July 29, 2013, 6:37am

I did look at the video. What do you mean?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 08:32, Kumar Surya Mani ksuryamani@gmail.com a écrit :

Hi David,

It sounds interesting; I would love it to be added to RSS River however I don't know whether you had a chance to look into that.

thanks & regards,
surya

On Mon, Jul 29, 2013 at 11:47 AM, David Pilato david@pilato.fr wrote:

We could add your video to RSS River page:
https://github.com/dadoonet/rssriver/blob/gh-pages/index.html
Or
https://github.com/dadoonet/rssriver/blob/master/README.md

What do you think? Open a pull request?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 06:43, Kumar Surya Mani ksuryamani@gmail.com a écrit :

I have uploaded a video for RSS river mapping, indexing and searching example; you can watch it here http://www.youtube.com/watch?v=JaX70FpEkAA
It is only visual no audio.

thanks & regards,
surya

On Sunday, July 28, 2013 12:17:15 PM UTC+5:30, Nev Ada wrote:

I want to store RSS feeds in a searchable Elasticsearch store.

Using RSSRiver, I am confused as to what they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one, is the approach to write a script to generate /_mapping or should it just work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "VentureBeat"}]}})
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iBAjQ-jvbMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Warms n Regards,
Kumar Surya Mani

  **They conquer who believe they can. - John Dryden

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kumarsuryamani · July 29, 2013, 6:47am

David,
I simply meant whether it had required information or not in the video for
somebody to start, I was just curious after seeing comments from you.
Thank you so much for your time to look in that.

On Mon, Jul 29, 2013 at 12:07 PM, David Pilato david@pilato.fr wrote:

I did look at the video. What do you mean?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 08:32, Kumar Surya Mani ksuryamani@gmail.com a
écrit :

Hi David,

It sounds interesting; I would love it to be added to RSS River however I
don't know whether you had a chance to look into that.

thanks & regards,
surya

On Mon, Jul 29, 2013 at 11:47 AM, David Pilato david@pilato.fr wrote:

We could add your video to RSS River page:
https://github.com/dadoonet/rssriver/blob/gh-pages/index.html
Or
https://github.com/dadoonet/rssriver/blob/master/README.md

What do you think? Open a pull request?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 06:43, Kumar Surya Mani ksuryamani@gmail.com a
écrit :

I have uploaded a video for RSS river mapping, indexing and searching
example; you can watch it here
http://www.youtube.com/watch?v=JaX70FpEkAA
It is only visual no audio.

thanks & regards,
surya

On Sunday, July 28, 2013 12:17:15 PM UTC+5:30, Nev Ada wrote:

I want to store RSS feeds in a searchable Elasticsearch store.

Using RSSRiver http://www.pilato.fr/rssriver, I am confused as to
what they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one http://venturebeat.com/feed,
is the approach to write a script to generate /_mapping or should it
just work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "VentureBeat"**}]}})

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iBAjQ-jvbMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Warms n Regards,
Kumar Surya Mani
  **They conquer who believe they can. - John Dryden
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iBAjQ-jvbMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Warms n Regards,
Kumar Surya Mani

  **They conquer who believe they can. - John Dryden

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · July 29, 2013, 6:49am

It's a good tutorial IMHO. It could help others.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 08:47, Kumar Surya Mani ksuryamani@gmail.com a écrit :

David,
I simply meant whether it had required information or not in the video for somebody to start, I was just curious after seeing comments from you.
Thank you so much for your time to look in that.

On Mon, Jul 29, 2013 at 12:07 PM, David Pilato david@pilato.fr wrote:

I did look at the video. What do you mean?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 08:32, Kumar Surya Mani ksuryamani@gmail.com a écrit :

Hi David,

It sounds interesting; I would love it to be added to RSS River however I don't know whether you had a chance to look into that.

thanks & regards,
surya

On Mon, Jul 29, 2013 at 11:47 AM, David Pilato david@pilato.fr wrote:

We could add your video to RSS River page:
https://github.com/dadoonet/rssriver/blob/gh-pages/index.html
Or
https://github.com/dadoonet/rssriver/blob/master/README.md

What do you think? Open a pull request?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 juil. 2013 à 06:43, Kumar Surya Mani ksuryamani@gmail.com a écrit :

I have uploaded a video for RSS river mapping, indexing and searching example; you can watch it here http://www.youtube.com/watch?v=JaX70FpEkAA
It is only visual no audio.

thanks & regards,
surya

On Sunday, July 28, 2013 12:17:15 PM UTC+5:30, Nev Ada wrote:

I want to store RSS feeds in a searchable Elasticsearch store.

Using RSSRiver, I am confused as to what they want:

Do they require a /_mapping and a /_meta?

So given a nested RSS feed like this one, is the approach to write a script to generate /_mapping or should it just work?

Here's what I tried, but the server didn't seem to download anything ::

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "VentureBeat"}]}})
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iBAjQ-jvbMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Warms n Regards,
Kumar Surya Mani
  **They conquer who believe they can. - John Dryden
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iBAjQ-jvbMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Warms n Regards,
Kumar Surya Mani

  **They conquer who believe they can. - John Dryden

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ElasticSearch, RSS and `_mapping` requirement?

es.put('venture_beat/_meta', data={"type": "rss", "rss": {"feeds": [{"name": "venturebeat", "url" : "http://venturebeat.com/feed/"}]}})

es.put('venture_beat/_meta',
data={"type": "rss",
"rss": {"feeds": [{"name": "venturebeat",
"url" : "http://venturebeat.com/feed/"}]}})