# Streaming a large file

**URL:** <https://discuss.elastic.co/t/streaming-a-large-file/111957>\
**Category:** Elasticsearch\
**Created:** [December 15, 2017, 11:03am UTC](https://discuss.elastic.co/t/streaming-a-large-file/111957 "2017-12-15T11:03:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ESCoder](https://avatars.discourse-cdn.com/v4/letter/e/9e8a1a/32.png) [@ESCoder](https://discuss.elastic.co/u/ESCoder)\
**Post date:** [December 15, 2017, 11:03am UTC](https://discuss.elastic.co/t/streaming-a-large-file/111957/1 "2017-12-15T11:03:01Z")

</div>

hi, we are using ElasticSearch 5.6 and NEST, we are also using the ingest attachment plugin to do full-text content search of various documents like pdf, word etc.

For large files, potentially above 100 MB in size, how do we implement streaming so that we dont load the whole file in memory?  
can we use chunking in conjunction with the ingest processor?  
thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 15, 2017, 11:21am UTC](https://discuss.elastic.co/t/streaming-a-large-file/111957/2 "2017-12-15T11:21:48Z")

</div>

There is sadly nothing like that. You need to provide a full JSon document to elasticsearch.  
No streaming.

You can give a look at FSCrawler project which exposes a REST endpoint where you can directly upload a binary document. The good thing is that is running in an external process (not within elasticsearch node) so it will put much less pressure on your nodes.  
Warning: it’s a community tool, not maintained by elastic.

---

<div class="post-metadata">

**Author:** ![ESCoder](https://avatars.discourse-cdn.com/v4/letter/e/9e8a1a/32.png) [@ESCoder](https://discuss.elastic.co/u/ESCoder)\
**Post date:** [December 15, 2017, 2:53pm UTC](https://discuss.elastic.co/t/streaming-a-large-file/111957/3 "2017-12-15T14:53:44Z")

</div>

thanks for the reply dadoonet.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 12, 2018, 2:54pm UTC](https://discuss.elastic.co/t/streaming-a-large-file/111957/4 "2018-01-12T14:54:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.