# Just Pushed: Indexer Support + Twtiter indexer plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [September 20, 2010, 9:20pm UTC](https://discuss.elastic.co/t/just-pushed-indexer-support-twtiter-indexer-plugin/3349 "2010-09-20T21:20:50Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 20, 2010, 9:20pm UTC](https://discuss.elastic.co/t/just-pushed-indexer-support-twtiter-indexer-plugin/3349/1 "2010-09-20T21:20:50Z")
Hi, Both the indexer support and an initial indexer implementation for twitter sample stream are now at master. Index information can be found here: [http://github.com/elasticsearch/elasticsearch/issues/closed#issue/377](http://github.com/elasticsearch/elasticsearch/issues/closed#issue/377). The twitter indexer plugin here: [http://github.com/elasticsearch/elasticsearch/issues/closed#issue/378](http://github.com/elasticsearch/elasticsearch/issues/closed#issue/378). I encourage you to head over and read it. I had a great Eureka moment where I decided to store the indexers meta and state information as another index in elasticsearch (did someone say eating your own god food? 😉 ) which really means the indexer is very open and have a good state persistance api (the elasticsearch API). The indexer itself is pretty open, here is the twitter indexer (pretty simple): [http://github.com/elasticsearch/elasticsearch/blob/master/plugins/indexer/twitter/src/main/java/org/elasticsearch/indexer/twitter/TwitterIndexer.java](http://github.com/elasticsearch/elasticsearch/blob/master/plugins/indexer/twitter/src/main/java/org/elasticsearch/indexer/twitter/TwitterIndexer.java) . ``` Some of my plans are to provide for polygot indexers (write your own ``` indexer in groovy, ruby), but since it will probably require the elasticsearch Client API, I would love to first get proper support for the JVM lang (like it is with the groovy case). Also, I have some ideas for more indexers implementation like wikipedia, rabbitmq, JMS, redis, couchdb, and others. ``` As a side note, the sample twitter stream API is pretty slow (in ``` elasticsearch terms) so you can easily run it on your laptop without it breaking a sweat at all ;). Would have loved to test it with the firehose... . -shay.banon ---
**Author:** ![Sergio\_Bossa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sergio_bossa/32/3312_2.png) [@Sergio\_Bossa](https://discuss.elastic.co/u/Sergio_Bossa)\ **Post date:** [September 21, 2010, 8:04am UTC](https://discuss.elastic.co/t/just-pushed-indexer-support-twtiter-indexer-plugin/3349/2 "2010-09-21T08:04:34Z")
Hi Shay, I'm not sure I get the difference between indexing stuff by using the new indexer feature, or by normally sending data through client APIs (or curl or whatever): may you elaborate more? Thanks, Cheers, Sergio B. -- Sergio Bossa [http://www.linkedin.com/in/sergiob](http://www.linkedin.com/in/sergiob) ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [September 21, 2010, 8:13am UTC](https://discuss.elastic.co/t/just-pushed-indexer-support-twtiter-indexer-plugin/3349/3 "2010-09-21T08:13:56Z")
Shay can explain the details, but one thing that is very interesting for us is that it looks like indexer gives us automatic failover for the client. "Indexers are singletons within the cluster. They get allocated automatically to one of the nodes and run. If that node fails, an indexer will be automatically allocated to another node" When you send the data through client APIs, you're responsible for the client prg availability, etc. Indexer runs as part of the cluster, hence can have better availability. It can eliminate the code we have to monitor the client (our clients run continuously) Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Tue, Sep 21, 2010 at 11:04 AM, Sergio Bossa [sergio.bossa@gmail.com](mailto:sergio.bossa@gmail.com)wrote: > Hi Shay, > > I'm not sure I get the difference between indexing stuff by using the > new indexer feature, or by normally sending data through client APIs > (or curl or whatever): may you elaborate more? > > Thanks, > Cheers, > > Sergio B. > > -- > Sergio Bossa > [http://www.linkedin.com/in/sergiob](http://www.linkedin.com/in/sergiob) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 21, 2010, 8:31am UTC](https://discuss.elastic.co/t/just-pushed-indexer-support-twtiter-indexer-plugin/3349/4 "2010-09-21T08:31:40Z")
Hi, Berkay touched on an important point of this feature. It all boils down to how you index data into elasticsearch. There are two different categories of doing it, the first is by responding to real time events and applying them to the index. For example, indexing data as part of your "insert" path within the web application, or by using hooks into nosql solutions (like terrastore). The other is one where you need to write code that performs a bridge between the data coming in and the indexing API. As an example, writing something that listens on rabbitmq for indexing messages and applies them, or something that listens to the twitter stream and index it, or something that sits on top of couchdb \_changes and applies those, or even sit on top of a database and index it. For those cases, a custom runtime component is usually written to do this bridging. But then, when writing one, you need to handle its failover, and state (what I indexed last + other). The indexer support in elasticsearch aims at solving this problem. I did not name those two cases properly (need to think of better names) but I hope the examples make sense. -shay.banon On Tue, Sep 21, 2010 at 10:13 AM, Berkay Mollamustafaoglu \<[mberkay@gmail.com](mailto:mberkay@gmail.com) > wrote: > Shay can explain the details, but one thing that is very interesting for us > is that it looks like indexer gives us automatic failover for the client. > > "Indexers are singletons within the cluster. They get allocated > automatically to one of the nodes and run. If that node fails, an indexer > will be automatically allocated to another node" > > When you send the data through client APIs, you're responsible for the > client prg availability, etc. Indexer runs as part of the cluster, hence > can have better availability. It can eliminate the code we have to monitor > the client (our clients run continuously) > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Tue, Sep 21, 2010 at 11:04 AM, Sergio Bossa [sergio.bossa@gmail.com](mailto:sergio.bossa@gmail.com)wrote: > > > Hi Shay, > > > > I'm not sure I get the difference between indexing stuff by using the > > new indexer feature, or by normally sending data through client APIs > > (or curl or whatever): may you elaborate more? > > > > Thanks, > > Cheers, > > > > Sergio B. > > > > -- > > Sergio Bossa > > [http://www.linkedin.com/in/sergiob](http://www.linkedin.com/in/sergiob) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 21, 2010, 11:40am UTC](https://discuss.elastic.co/t/just-pushed-indexer-support-twtiter-indexer-plugin/3349/5 "2010-09-21T11:40:47Z")
One more thing, "indexer" has now been renamed to "river". So search/replace everything from indexer to river. Issues have been updated. On Tue, Sep 21, 2010 at 10:31 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Hi, > > Berkay touched on an important point of this feature. It all boils down > to how you index data into elasticsearch. There are two different categories > of doing it, the first is by responding to real time events and applying > them to the index. For example, indexing data as part of your "insert" path > within the web application, or by using hooks into nosql solutions (like > terrastore). > > The other is one where you need to write code that performs a bridge > between the data coming in and the indexing API. As an example, writing > something that listens on rabbitmq for indexing messages and applies them, > or something that listens to the twitter stream and index it, or something > that sits on top of couchdb \_changes and applies those, or even sit on top > of a database and index it. For those cases, a custom runtime component is > usually written to do this bridging. But then, when writing one, you need to > handle its failover, and state (what I indexed last + other). The indexer > support in elasticsearch aims at solving this problem. > > I did not name those two cases properly (need to think of better names) > but I hope the examples make sense. > > -shay.banon > > On Tue, Sep 21, 2010 at 10:13 AM, Berkay Mollamustafaoglu \< > [mberkay@gmail.com](mailto:mberkay@gmail.com)\> wrote: > > > Shay can explain the details, but one thing that is very interesting for > > us is that it looks like indexer gives us automatic failover for the > > client. > > > > "Indexers are singletons within the cluster. They get allocated > > automatically to one of the nodes and run. If that node fails, an indexer > > will be automatically allocated to another node" > > > > When you send the data through client APIs, you're responsible for the > > client prg availability, etc. Indexer runs as part of the cluster, hence > > can have better availability. It can eliminate the code we have to monitor > > the client (our clients run continuously) > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Tue, Sep 21, 2010 at 11:04 AM, Sergio Bossa [sergio.bossa@gmail.com](mailto:sergio.bossa@gmail.com)wrote: > > > > > Hi Shay, > > > > > > I'm not sure I get the difference between indexing stuff by using the > > > new indexer feature, or by normally sending data through client APIs > > > (or curl or whatever): may you elaborate more? > > > > > > Thanks, > > > Cheers, > > > > > > Sergio B. > > > > > > -- > > > Sergio Bossa > > > [http://www.linkedin.com/in/sergiob](http://www.linkedin.com/in/sergiob) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:19am UTC](https://discuss.elastic.co/t/just-pushed-indexer-support-twtiter-indexer-plugin/3349/6 "2017-07-06T04:19:10Z")
Closed on July 6, 2017, 4:19am UTC