# River index by date **URL:** \ **Category:** Elasticsearch\ **Created:** [January 15, 2013, 10:11pm UTC](https://discuss.elastic.co/t/river-index-by-date/10361 "2013-01-15T22:11:43Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Martin\_Janda](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@Martin\_Janda](https://discuss.elastic.co/u/Martin_Janda)\ **Post date:** [January 15, 2013, 10:11pm UTC](https://discuss.elastic.co/t/river-index-by-date/10361/1 "2013-01-15T22:11:43Z")
Hi, I'm trying to index tweets stored in CouchDB with a river plugin (I'm starting with CouchDB full of tweets and brand new empty ES server). That works fine but I also need to index them by date. So if a tweet has a created\_at attribute "Mon, 14 Jan 2013 02:22:33 +0000", I'd like to index it under 2013-01-14 index/type. I though the easy solution might be to use a "script" attribute when creating the index, where I'd set the ctx.\_type attribute to a dynamically computed value, e.g. '2013-01-14'. I've been trying that for quite a long time without luck. The idea should be apparent from the following river index definition: { "type": "couchdb", "couchdb": { "host": "localhost", "port": "5984", "db": "my\_db", "user": "user", "password": "password", "filter": null, "script": "here I put javascript commands; that compute the type name from ctx.doc.created\_at (var computed\_type\_name=...) and then set it as follows; ctx.\_type=computed\_type\_name;" }, "index": { "index": "my\_index" } } I put some semicolons there to suggest that I have multiple commands there. When I PUT this JSON into ES with "script": null, it indexes all documents. When I put there the example shown above, it creates index and type my\_index, which stays empty. I get no error. This thread[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx.\_type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx._type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ)lead me to belive this should work. I also tried a naive solution with "script": "ctx.\_type=ctx.doc.created\_at" but that also didn't index anything. My question is, should I expect this to work or is there a better approach? I need to keep alive only ~7 last couple of days, no need to search through too old tweets. -- ---
**Author:** ![Michael\_Jackson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_jackson/32/2537_2.png) [@Michael\_Jackson](https://discuss.elastic.co/u/Michael_Jackson)\ **Post date:** [January 15, 2013, 10:26pm UTC](https://discuss.elastic.co/t/river-index-by-date/10361/2 "2013-01-15T22:26:12Z")
Not sure this is the problem, but did you install the lang-javascript[https://github.com/elasticsearch/elasticsearch-lang-javascript](https://github.com/elasticsearch/elasticsearch-lang-javascript)plugin? ES needs it to run your script snippet. $ bin/plugin -install elasticsearch/elasticsearch-river-couchdb/1.1.0 -- Michael Jackson @mjackson On Tue, Jan 15, 2013 at 2:11 PM, Martin Janda [damdrdidlda@gmail.com](mailto:damdrdidlda@gmail.com) wrote: > Hi, > > I'm trying to index tweets stored in CouchDB with a river plugin (I'm > starting with CouchDB full of tweets and brand new empty ES server). That > works fine but I also need to index them by date. So if a tweet has a > created\_at attribute "Mon, 14 Jan 2013 02:22:33 +0000", I'd like to index > it under 2013-01-14 index/type. I though the easy solution might be to use > a "script" attribute when creating the index, where I'd set the ctx.\_type > attribute to a dynamically computed value, e.g. '2013-01-14'. I've been > trying that for quite a long time without luck. The idea should be apparent > from the following river index definition: > > { > "type": "couchdb", > "couchdb": { > "host": "localhost", > "port": "5984", > "db": "my\_db", > "user": "user", > "password": "password", > "filter": null, > "script": "here I put javascript commands; that compute the type > name from ctx.doc.created\_at (var computed\_type\_name=...) and then set it > as follows; ctx.\_type=computed\_type\_name;" > }, > "index": { > "index": "my\_index" > } > } > > I put some semicolons there to suggest that I have multiple commands > there. When I PUT this JSON into ES with "script": null, it indexes all > documents. When I put there the example shown above, it creates index and > type my\_index, which stays empty. I get no error. > This thread[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx.\_type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx._type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ)lead me to belive this should work. I also tried a naive solution > with "script": "ctx.\_type=ctx.doc.created\_at" but that also didn't index > anything. > > My question is, should I expect this to work or is there a better > approach? I need to keep alive only ~7 last couple of days, no need to > search through too old tweets. > > -- -- ---
**Author:** ![Martin\_Janda](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@Martin\_Janda](https://discuss.elastic.co/u/Martin_Janda)\ **Post date:** [January 15, 2013, 10:28pm UTC](https://discuss.elastic.co/t/river-index-by-date/10361/3 "2013-01-15T22:28:09Z")
Yes, I have. This version: elasticsearch-lang-javascript-1.2.0. My river plugin version matches with yours. On Tuesday, 15 January 2013 23:26:12 UTC+1, Michael Jackson wrote: > Not sure this is the problem, but did you install the lang-javascript[https://github.com/elasticsearch/elasticsearch-lang-javascript](https://github.com/elasticsearch/elasticsearch-lang-javascript)plugin? ES needs it to run your script snippet. > > $ bin/plugin -install elasticsearch/elasticsearch-river-couchdb/1.1.0 > > -- > Michael Jackson > @mjackson > > On Tue, Jan 15, 2013 at 2:11 PM, Martin Janda \<[damdr...@gmail.com](mailto:damdr...@gmail.com)\ > > > wrote: > > > Hi, > > > > I'm trying to index tweets stored in CouchDB with a river plugin (I'm > > starting with CouchDB full of tweets and brand new empty ES server). That > > works fine but I also need to index them by date. So if a tweet has a > > created\_at attribute "Mon, 14 Jan 2013 02:22:33 +0000", I'd like to index > > it under 2013-01-14 index/type. I though the easy solution might be to use > > a "script" attribute when creating the index, where I'd set the ctx.\_type > > attribute to a dynamically computed value, e.g. '2013-01-14'. I've been > > trying that for quite a long time without luck. The idea should be apparent > > from the following river index definition: > > > > { > > "type": "couchdb", > > "couchdb": { > > "host": "localhost", > > "port": "5984", > > "db": "my\_db", > > "user": "user", > > "password": "password", > > "filter": null, > > "script": "here I put javascript commands; that compute the type > > name from ctx.doc.created\_at (var computed\_type\_name=...) and then set > > it as follows; ctx.\_type=computed\_type\_name;" > > }, > > "index": { > > "index": "my\_index" > > } > > } > > > > I put some semicolons there to suggest that I have multiple commands > > there. When I PUT this JSON into ES with "script": null, it indexes all > > documents. When I put there the example shown above, it creates index and > > type my\_index, which stays empty. I get no error. > > This thread[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx.\_type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx._type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ)lead me to belive this should work. I also tried a naive solution > > with "script": "ctx.\_type=ctx.doc.created\_at" but that also didn't index > > anything. > > > > My question is, should I expect this to work or is there a better > > approach? I need to keep alive only ~7 last couple of days, no need to > > search through too old tweets. > > > > -- -- ---
**Author:** ![Martin\_Janda](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@Martin\_Janda](https://discuss.elastic.co/u/Martin_Janda)\ **Post date:** [February 7, 2013, 11:33pm UTC](https://discuss.elastic.co/t/river-index-by-date/10361/4 "2013-02-07T23:33:13Z")
I found a satisfying solution: elasticsearch-rollindex[https://github.com/karussell/elasticsearch-rollindex](https://github.com/karussell/elasticsearch-rollindex) plugin. The idea is to create a rolling index that will roll each day. It always creates an index with unique name like index\_2013-02-05-00-00. The important thing is that the most recent index always has a constant alias index\_feed (rollindex takes care of that). Then I create a river that streams data into the index\_feed. In order to make it work, I had to supply a dummy\_index for the \_river plugin to redirect data into index\_feed via a script as follows: # you have to set up a cron to run this periodically curl -XPUT "localhost:9200/\_rollindex?indexPrefix=index&searchIndices=1&rollIndices=7&closeAfterRoll=false # this needs to be run just once curl -XPUT localhost:9200/\_river/dummy\_index/\_meta -d '{ "type": "couchdb", "couchdb": { "host": "localhost", "port": "5984", "db": "my\_couchdb\_database", "user": "username", "password": "password", "filter": null, "script": "ctx.\_index="index\_feed"" }, "index": { "index": "dummy\_index", "type": "some\_type", "bulk\_size": "100", "bulk\_timeout": "10ms" } }' I hope someone will find this information useful. If you know about a more elegant solution, please let me know. On Tuesday, 15 January 2013 23:28:09 UTC+1, Martin Janda wrote: > Yes, I have. This version: elasticsearch-lang-javascript-1.2.0. My river > plugin version matches with yours. > > On Tuesday, 15 January 2013 23:26:12 UTC+1, Michael Jackson wrote: > > > Not sure this is the problem, but did you install the lang-javascript[https://github.com/elasticsearch/elasticsearch-lang-javascript](https://github.com/elasticsearch/elasticsearch-lang-javascript)plugin? ES needs it to run your script snippet. > > > > $ bin/plugin -install elasticsearch/elasticsearch-river-couchdb/1.1.0 > > > > -- > > Michael Jackson > > @mjackson > > > > On Tue, Jan 15, 2013 at 2:11 PM, Martin Janda [damdr...@gmail.com](mailto:damdr...@gmail.com) wrote: > > > > > Hi, > > > > > > I'm trying to index tweets stored in CouchDB with a river plugin (I'm > > > starting with CouchDB full of tweets and brand new empty ES server). That > > > works fine but I also need to index them by date. So if a tweet has a > > > created\_at attribute "Mon, 14 Jan 2013 02:22:33 +0000", I'd like to index > > > it under 2013-01-14 index/type. I though the easy solution might be to use > > > a "script" attribute when creating the index, where I'd set the ctx.\_type > > > attribute to a dynamically computed value, e.g. '2013-01-14'. I've been > > > trying that for quite a long time without luck. The idea should be apparent > > > from the following river index definition: > > > > > > { > > > "type": "couchdb", > > > "couchdb": { > > > "host": "localhost", > > > "port": "5984", > > > "db": "my\_db", > > > "user": "user", > > > "password": "password", > > > "filter": null, > > > "script": "here I put javascript commands; that compute the type > > > name from ctx.doc.created\_at (var computed\_type\_name=...) and then set > > > it as follows; ctx.\_type=computed\_type\_name;" > > > }, > > > "index": { > > > "index": "my\_index" > > > } > > > } > > > > > > I put some semicolons there to suggest that I have multiple commands > > > there. When I PUT this JSON into ES with "script": null, it indexes all > > > documents. When I put there the example shown above, it creates index and > > > type my\_index, which stays empty. I get no error. > > > This thread[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx.\_type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/ctx._type/elasticsearch/S0bmeMvnLyQ/Pla3UShs6fgJ)lead me to belive this should work. I also tried a naive solution > > > with "script": "ctx.\_type=ctx.doc.created\_at" but that also didn't index > > > anything. > > > > > > My question is, should I expect this to work or is there a better > > > approach? I need to keep alive only ~7 last couple of days, no need to > > > search through too old tweets. > > > > > > -- -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:52am UTC](https://discuss.elastic.co/t/river-index-by-date/10361/5 "2017-07-06T02:52:30Z")