We have been experiencing what feels like rather poor performance for our percolate queries. We have 4 indexes with percolate queries, each with a few hundred saved queries. These indexes have a single shard and many replicas (to attempt to spread out the load from the percolate queries). Our cluster is running ES 6.8 and has 27 data nodes each 16 CPUs, 52gb RAM, 2.8TB NVME storage. When running percolate queries, the highest throughput we have been able to achieve is 1k to 2k documents percolated per second. For perspective, we regularly easily index over 10k documents a second (the limits are generally from other processing systems, not the ES cluster), and for searching we have seen some of our indexes peak at over 100k searches a second. In general, outside of percolate, the cluster generally has acceptable performance.
I was wondering what the best practices were when structuring indexes for maximum percolate performance. Should creating a replica for each node scale performance linearly? Should we dedicate some nodes to just holding the percolate indexes? We have some dynamic templates for defining the mapping of of the documents to be percolated, should we specify those in the mapping of the index rather than using the dynamic template? What are the factors that generally slow down percolate performance?
Thank you for any help or guidance.