I'm currently writing an application that heavily uses ES percolator. To give an idea about what I'm trying to do:
I'm percolating an average of 20k new documents through a job that runs every 2 mins, each document is 5 KB on average, against 10k+ queries. The goal is to have this job finish executing in less than 2 minutes, but now (with the following cluster setup) it's taking not less than 15 minutes to finish.
I have a dedicated cluster for the percolator index to live, away from the main data cluster, with 6 c3.xlarge nodes, with 10 shards and replica 1.
Any ideas/suggestions for improving the current performance? should adding more nodes solve the problem?