# \[ANN\] FSRiver 0.3.0 for elasticsearch 0.90.3 **URL:** \ **Category:** Elasticsearch\ **Created:** [August 9, 2013, 3:30pm UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152 "2013-08-09T15:30:46Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 9, 2013, 3:30pm UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/1 "2013-08-09T15:30:46Z")
Heya, I just released File System River (aka FSRiver) 0.3.0: [https://github.com/dadoonet/fsriver](https://github.com/dadoonet/fsriver) This version contains: Add SSH support Update to Elasticsearch 0.90.3 / Mapper Attachment 1.8.0 Use BulkProcessor feature Issues: [https://github.com/dadoonet/fsriver/issues?milestone=4&state=closed](https://github.com/dadoonet/fsriver/issues?milestone=4&state=closed) PR, feedbacks, comments are welcome 😉 -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs ---
**Author:** ![Fatima\_Castiglione\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fatima_castiglione_m/32/2252_2.png) [@Fatima\_Castiglione\_M](https://discuss.elastic.co/u/Fatima_Castiglione_M)\ **Post date:** [August 9, 2013, 5:41pm UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/2 "2013-08-09T17:41:31Z")
Hi Well done Do you think this would fix ScrutMyDocs automatic indexing? Have you tried? Thanks in advance 2013/8/9 David Pilato [david@pilato.fr](mailto:david@pilato.fr) > Heya, > > I just released File System River (aka FSRiver) 0.3.0: > [GitHub - dadoonet/fscrawler: Elasticsearch File System Crawler (FS Crawler)](https://github.com/dadoonet/fsriver) > > This version contains: > > - Add SSH support > - Update to Elasticsearch 0.90.3 / Mapper Attachment 1.8.0 > - Use BulkProcessor feature > > Issues: > [GitHub · Where software is built](https://github.com/dadoonet/fsriver/issues?milestone=4&state=closed) > > PR, feedbacks, comments are welcome 😉 > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) # -- Fátima Castiglione Maldonado [castiglionemaldonado@gmail.com](mailto:castiglionemaldonado@gmail.com) ``` ____ ,'_ | ``` **_||| \< ) .------. `-----------,------.-' ,-' `-. | | | ,' `. ,' | | ,' `. | ,-' |_ / _,'-' `. `---._|**\_\_\_\_\_\_\_\_\_ .--' -----. | \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ `-. ----- | | ___| | | \ ,- \ | |___ | |===========================((|) | | | | | | _____________________ / `- / | `--._ -----' | _________________ ,-' ----- | `.-.\_ ,' \_\_.---' | / | `-. | \ / `. | | `. ,' | | | `. ,' \_\_\_\_\_,----------`-------`-. `-._ _,-' < ___________________________ ) `------' | _| | `._\_\_\_| ================================= -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pellyadolfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pellyadolfo/32/1879_2.png) [@pellyadolfo](https://discuss.elastic.co/u/pellyadolfo)\ **Post date:** [November 20, 2013, 3:15am UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/3 "2013-11-20T03:15:57Z")
I am a new to ES so sorry if this is a trivial question. I have some JSON files already in my file system. By using fsRiver to index them into ES, will they be replicated somewhere in the hard drive or ES just creates and stores only indices for the these original JSON documents (so they are not replicated)? Thanks (My documents are instead XML docs but conversion into JSON is not a problem, if fs River cannot handle, I will write some code, but I hate the idea of having the documents replicated again by ES in the harddrive). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 20, 2013, 3:20am UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/4 "2013-11-20T03:20:45Z")
ES will store the source within it's own data store (ie on disk), however it does do compression. This might help explain things - > Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 20 November 2013 14:15, [pellyadolfo@yahoo.es](mailto:pellyadolfo@yahoo.es) wrote: > I am a new to ES so sorry if this is a trivial question. > > I have some JSON files already in my file system. By using fsRiver to > index them into ES, will they be replicated somewhere in the hard drive or > ES just creates and stores only indices for the these original JSON > documents (so they are not replicated)? > > Thanks > > (My documents are instead XML docs but conversion into JSON is not a > problem, if fs River cannot handle, I will write some code, but I hate the > idea of having the documents replicated again by ES in the harddrive). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pellyadolfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pellyadolfo/32/1879_2.png) [@pellyadolfo](https://discuss.elastic.co/u/pellyadolfo)\ **Post date:** [November 20, 2013, 3:49am UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/5 "2013-11-20T03:49:53Z")
Thanks Marc, definitely it helps. This is a related thread. [https://groups.google.com/forum/#!topicsearchin/elasticsearch/store$20json|sort:date/elasticsearch/ms3A6vWTAVk](https://groups.google.com/forum/#!topicsearchin/elasticsearch/store%2420json%7Csort:date/elasticsearch/ms3A6vWTAVk) You can configure if the field is stored and/or indexed but in any case the original JSON document is stored compressed by ES (as you said). What I would be missing in ES is an option to say ES: "do not store the JSON document and take in from here instead whenever is needed" (optionally writing a kind of provider/format converter) It clarifies, thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 20, 2013, 4:01am UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/6 "2013-11-20T04:01:19Z")
The only issue with making it read from an external source is that you're adding in extra delays and potential points of failure. You can do this but it'd be a custom script. I understand the desire to increase efficiencies in disk use, but the compression for ES is very good. eg We get around 50% compression with logging data. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 20 November 2013 14:49, [pellyadolfo@yahoo.es](mailto:pellyadolfo@yahoo.es) wrote: > Thanks Marc, definitely it helps. > > This is a related thread. > [Redirecting to Google Groups](https://groups.google.com/forum/#!topicsearchin/elasticsearch/store$20json%7Csort:date/elasticsearch/ms3A6vWTAVk) > > You can configure if the field is stored and/or indexed but in any case > the original JSON document is stored compressed by ES (as you said). > > What I would be missing in ES is an option to say ES: "do not store the > JSON document and take in from here instead whenever is needed" (optionally > writing a kind of provider/format converter) > > It clarifies, thanks > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 20, 2013, 3:30pm UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/7 "2013-11-20T15:30:43Z")
You could also think of disabling \_all and \_source fields. [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html#mapping-all-field](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html#mapping-all-field) [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html#mapping-source-field](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html#mapping-source-field) -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 20 novembre 2013 at 05:01:42, Mark Walkom ([markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)) a écrit: The only issue with making it read from an external source is that you're adding in extra delays and potential points of failure. You can do this but it'd be a custom script. I understand the desire to increase efficiencies in disk use, but the compression for ES is very good. eg We get around 50% compression with logging data. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 20 November 2013 14:49, [pellyadolfo@yahoo.es](mailto:pellyadolfo@yahoo.es) wrote: Thanks Marc, definitely it helps. This is a related thread. [https://groups.google.com/forum/#!topicsearchin/elasticsearch/store$20json|sort:date/elasticsearch/ms3A6vWTAVk](https://groups.google.com/forum/#!topicsearchin/elasticsearch/store%2420json%7Csort:date/elasticsearch/ms3A6vWTAVk) You can configure if the field is stored and/or indexed but in any case the original JSON document is stored compressed by ES (as you said). What I would be missing in ES is an option to say ES: "do not store the JSON document and take in from here instead whenever is needed" (optionally writing a kind of provider/format converter) ## It clarifies, thanks You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:05am UTC](https://discuss.elastic.co/t/ann-fsriver-0-3-0-for-elasticsearch-0-90-3/13152/8 "2017-07-06T02:05:55Z")