[hadoop] control of inputsplit number

Hello there.

I'm working with the Hadoop-ElasticSearch jar to distribute an algorithm
over a sharded index.
The processing is fairly simple : performing a given task on each document
inside the index.

As the algorithm only works on one document, it can be distributed and
Therefore I put it in a MapReduce job, within the mapper. No reducer is

Currently, the index has 6 shard, and consequently, my algorithm is run in
parallel in 6 different mappers.
But the index contains about 3 millions documents, and each map process
(sequentially) about 500 000 documents, which takes 4 hour in average.

I'd like to have more mappers executed in parallel.

I read on the documentation that associating an InputSplit per Shard is
done in purpose. I can change the number of shard, but I wonder if there
was an alternative.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c862573f-57e1-4bb5-8d84-494a5de163df%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.