# \[hadoop\] control of inputsplit number

**URL:** <https://discuss.elastic.co/t/hadoop-control-of-inputsplit-number/16172>\
**Category:** Elasticsearch\
**Created:** [March 5, 2014, 12:49pm UTC](https://discuss.elastic.co/t/hadoop-control-of-inputsplit-number/16172 "2014-03-05T12:49:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Damien\_Feugas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/damien_feugas/32/1750_2.png) [@Damien\_Feugas](https://discuss.elastic.co/u/Damien_Feugas)\
**Post date:** [March 5, 2014, 12:49pm UTC](https://discuss.elastic.co/t/hadoop-control-of-inputsplit-number/16172/1 "2014-03-05T12:49:28Z")

</div>

Hello there.

I'm working with the Hadoop-ElasticSearch jar to distribute an algorithm  
over a sharded index.  
The processing is fairly simple : performing a given task on each document  
inside the index.

As the algorithm only works on one document, it can be distributed and  
parallelised.  
Therefore I put it in a MapReduce job, within the mapper. No reducer is  
needed.

Currently, the index has 6 shard, and consequently, my algorithm is run in  
parallel in 6 different mappers.  
But the index contains about 3 millions documents, and each map process  
(sequentially) about 500 000 documents, which takes 4 hour in average.

I'd like to have more mappers executed in parallel.

I read on the documentation that associating an InputSplit per Shard is  
done in purpose. I can change the number of shard, but I wonder if there  
was an alternative.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c862573f-57e1-4bb5-8d84-494a5de163df%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c862573f-57e1-4bb5-8d84-494a5de163df%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:45am UTC](https://discuss.elastic.co/t/hadoop-control-of-inputsplit-number/16172/2 "2017-07-06T01:45:36Z")

</div>

Closed on July 6, 2017, 1:45am UTC