# Specify 'preference' query param in spark

**URL:** <https://discuss.elastic.co/t/specify-preference-query-param-in-spark/66308>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [November 16, 2016, 9:11pm UTC](https://discuss.elastic.co/t/specify-preference-query-param-in-spark/66308 "2016-11-16T21:11:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mgreene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mgreene/32/13216_2.png) [@mgreene](https://discuss.elastic.co/u/mgreene)\
**Post date:** [November 16, 2016, 9:11pm UTC](https://discuss.elastic.co/t/specify-preference-query-param-in-spark/66308/1 "2016-11-16T21:11:47Z")

</div>

I was wondering if there was a way to specify the `preference` query param when initializing an RDD via `JavaEsSpark.esRDD(sparkContext, "indexName/docType", esQuery);`?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [November 18, 2016, 5:20pm UTC](https://discuss.elastic.co/t/specify-preference-query-param-in-spark/66308/2 "2016-11-18T17:20:27Z")

</div>

At the moment there is no real way to do this. ES-Hadoop makes heavy use of the preference parameter as it is, so I'm interested in hearing your use case to see if it's something we can support better.

---

<div class="post-metadata">

**Author:** ![mgreene](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mgreene/32/13216_2.png) [@mgreene](https://discuss.elastic.co/u/mgreene)\
**Post date:** [November 18, 2016, 9:18pm UTC](https://discuss.elastic.co/t/specify-preference-query-param-in-spark/66308/3 "2016-11-18T21:18:47Z")

</div>

Hey James,

In my tests where I was monitoring network throughput on 15 data nodes, when the RDD starts streaming data, it appears to rely on a single coordinator node for the entire query to funnel data back into the Spark RDD. This proves to be a bottleneck for even modestly sized Spark clusters on modern hardware.

I ended up writing my own custom RDD that discovers all of the shards for a given index and launches the query independently to each shard with the preference parameter set. I found this approach to be about 30-40% faster in streaming throughput for my job which pulled about 400 million documents totaling about 10GB.

I'm using ES 2.4.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2016, 9:18pm UTC](https://discuss.elastic.co/t/specify-preference-query-param-in-spark/66308/4 "2016-12-16T21:18:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.