# Accelerate search scroll? **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [October 14, 2016, 2:39pm UTC](https://discuss.elastic.co/t/accelerate-search-scroll/63064 "2016-10-14T14:39:18Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [October 14, 2016, 2:39pm UTC](https://discuss.elastic.co/t/accelerate-search-scroll/63064/1 "2016-10-14T14:39:18Z")
Hello, I want to export elasticsearch index to HDFS, via let's say PIG. It's working fine, but it's slow, index with 3 shards, cluster with 3 nodes BUT a single client node (access point). Before, I was using a small home made Java program that creates X workers (multi-thread) for each shard that search-scroll in same time, took 10 minutes to load the entire index. Whereas with hadoop, I can see there is a single search-scroll process, and it took 30 minutes 😕 Is it possible to "force" the number of workers, even with a single client node? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [October 14, 2016, 6:21pm UTC](https://discuss.elastic.co/t/accelerate-search-scroll/63064/2 "2016-10-14T18:21:59Z")
@ebuildy do you see the same sort of bottlenecking when using a different framework other than Pig? I suspect that this has to do with Pig's input split combining. By default, Pig tries to combine multiple input splits into one map task. When ES-Hadoop creates the input splits, there's no way for us to accurately estimate the size of the data within a shard, so the input splits we return are of size `1`. This can cause Pig to think that many of the splits can be easily read by a single map task. I would try running the job again but setting the `pig.noSplitCombination` property to false. ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [October 15, 2016, 10:28am UTC](https://discuss.elastic.co/t/accelerate-search-scroll/63064/3 "2016-10-15T10:28:28Z")
You are right, I see some improvements with > SET pig.noSplitCombination TRUE; > SET default\_parallel 5; And I can see 1 open\_contexts by shard! Thanks you. I am trying with 3 shards to see if there are 3 open\_contexts. ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [October 15, 2016, 12:20pm UTC](https://discuss.elastic.co/t/accelerate-search-scroll/63064/4 "2016-10-15T12:20:07Z")
Now I am trying with Hive, and see a single search\_contexts 😕 by default Here [http://b3.ms/XmWn0WY2AYOk](http://b3.ms/XmWn0WY2AYOk) On left, Pig =\> 100Mbps On right, Hive =\> 30 Mbps Let's tune it! ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [October 17, 2016, 10:11am UTC](https://discuss.elastic.co/t/accelerate-search-scroll/63064/5 "2016-10-17T10:11:11Z")
No chance to make it working better 😕 I see there is some changes on v5.0.0 that implement parrarel multi - shards right? For now it's really to slow to work with 😕 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:22pm UTC](https://discuss.elastic.co/t/accelerate-search-scroll/63064/6 "2017-07-06T13:22:53Z")