# \[HOWTO\] Creating a streaming JSON river

**URL:** <https://discuss.elastic.co/t/howto-creating-a-streaming-json-river/9110>\
**Category:** Elasticsearch\
**Created:** [September 23, 2012, 1:57pm UTC](https://discuss.elastic.co/t/howto-creating-a-streaming-json-river/9110 "2012-09-23T13:57:37Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [September 23, 2012, 1:57pm UTC](https://discuss.elastic.co/t/howto-creating-a-streaming-json-river/9110/1 "2012-09-23T13:57:37Z")

</div>

Hi there,

I have created a sample small river (intended to help you write your  
own implementation, not to use this one), which can be used to stream  
data into elasticsearch via JSON. Basically this does not sound to  
impressive, but let me tell first

- It really streams data in (from any HTTP endpoint), this means, you  
can import gigabytes of JSON data without going OOM as you do not need  
to hold the whole response in memory
- Its architecture is to be working incrementally, as it provides a  
timestamp to the endpoint, which then only should deliver the data  
which has updates - you need to implement this logic in your endpoint  
of course as well
- Even though it uses bulk imports, it has a max\_bulk\_size which  
means, the river fires a bulk import every n\*thousand products making  
them searchable even before your data import is finished
- Comes with sample tests, which might help you to test your own river  
implementation
- NOT implemented, but mentioned in the documentation: Extending your  
river to read its configuration from the river configuration instead  
of being hardcoded in the source.

You can get the source and its documentation (I tried to make it  
extensive) at [https://github.com/spinscale/elasticsearch-river-streaming-json](https://github.com/spinscale/elasticsearch-river-streaming-json)

Have fun, create your own river and, as usual, drop some feedback.  
And yes, I know, its not always desired to import products, before the  
response has been coming in, but I think in many cases this works  
pretty. Always think about your use-case.

Regards, Alexander

P.S. I also created a small pull request to get the documentation for  
this into the ES tutorials page  
([https://github.com/elasticsearch/elasticsearch.github.com/pull/300](https://github.com/elasticsearch/elasticsearch.github.com/pull/300))

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:11am UTC](https://discuss.elastic.co/t/howto-creating-a-streaming-json-river/9110/2 "2017-07-06T03:11:44Z")

</div>