# Low performance of Spark streaming with elastic search

**URL:** <https://discuss.elastic.co/t/low-performance-of-spark-streaming-with-elastic-search/73609>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [February 2, 2017, 1:27am UTC](https://discuss.elastic.co/t/low-performance-of-spark-streaming-with-elastic-search/73609 "2017-02-02T01:27:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Johnnie](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Johnnie](https://discuss.elastic.co/u/Johnnie)\
**Post date:** [February 2, 2017, 1:27am UTC](https://discuss.elastic.co/t/low-performance-of-spark-streaming-with-elastic-search/73609/1 "2017-02-02T01:27:51Z")

</div>

Hi,

Our spark streaming user case is, read streaming data from Kafka, and join with index from Elastic Search.  
There is another spark streaming job update the Elastic Search index at a fix interval, which means the index data is not static.

Platform details is as below.  
Spark 1.6.2 standalone cluster with 15 nodes. 90 cores 500G memory.  
Elastic Search 2.4 with 10 data nodes. about 1Billion documents in Elastic search about 250G. 20 shards.

Our interval is 1hour and the job read the whole index data from Elastic search is 50 minutes.

is there any way to improve the performance of the job?

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 2, 2017, 1:28am UTC](https://discuss.elastic.co/t/low-performance-of-spark-streaming-with-elastic-search/73609/2 "2017-03-02T01:28:16Z")

</div>

Closed on March 2, 2017, 1:28am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.