# Percolator performance ideas **URL:** \ **Category:** Elasticsearch\ **Created:** [June 8, 2012, 11:00am UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019 "2012-06-08T11:00:40Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jschelle\_2](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@jschelle\_2](https://discuss.elastic.co/u/jschelle_2)\ **Post date:** [June 8, 2012, 11:00am UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019/1 "2012-06-08T11:00:40Z")
I'm trying to optimize the performance of my percolators, and hoping folks out there have some ideas, as I can't find a lot out there. Percolator performance seems to be great until I hit ~100,000 queries, and then falls off. Unfortunately I need to be able to support 500,000-1,000,000 queries per index. Queries are already pretty optimized. I'm using a sample query for now anyway. I do use the query property to filter which queries get executed. I tried turning off auto\_expand\_replica but it seemed to ignore me. I manually create the \_percolator index and set it to 2 shards and 0 replicas but it instantly sets replicas to 1. Can the \_percolator index be shared like a regular index? And will the engine distribute the percolating document across shards like the search side of things? Can Routing be used with percolate? Any other ideas?? Thanks in advance! ---
**Author:** ![jschelle\_2](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@jschelle\_2](https://discuss.elastic.co/u/jschelle_2)\ **Post date:** [June 8, 2012, 1:16pm UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019/2 "2012-06-08T13:16:54Z")
TYPO: Can the \_percolator index be _sharded_ like a regular index? On Friday, June 8, 2012 7:00:40 AM UTC-4, shadow000fire wrote: > I'm trying to optimize the performance of my percolators, and hoping folks > out there have some ideas, as I can't find a lot out there. Percolator > performance seems to be great until I hit ~100,000 queries, and then falls > off. Unfortunately I need to be able to support 500,000-1,000,000 queries > per index. > > Queries are already pretty optimized. I'm using a sample query for now > anyway. > > I do use the query property to filter which queries get executed. > > I tried turning off auto\_expand\_replica but it seemed to ignore me. > > I manually create the \_percolator index and set it to 2 shards and 0 > replicas but it instantly sets replicas to 1. Can the \_percolator index be > shared like a regular index? And will the engine distribute the percolating > document across shards like the search side of things? > > Can Routing be used with percolate? > > Any other ideas?? > > Thanks in advance! ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 11, 2012, 2:54pm UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019/3 "2012-06-11T14:54:33Z")
The design of the \_percolator is such that it needs to exist on all nodes storage wise. It loads the relevant queries when actual indices are allocated. I have some ideas on how to improve that, but for now, you can do the partitioning yourself by having some queries in one "index" (i.e. /\_percolator/index1) and another in another index (/\_percolator/index2), and execute the percolate request on both. On Fri, Jun 8, 2012 at 1:00 PM, Jason Scheller [jason.scheller@gmail.com](mailto:jason.scheller@gmail.com)wrote: > I'm trying to optimize the performance of my percolators, and hoping folks > out there have some ideas, as I can't find a lot out there. Percolator > performance seems to be great until I hit ~100,000 queries, and then falls > off. Unfortunately I need to be able to support 500,000-1,000,000 queries > per index. > > Queries are already pretty optimized. I'm using a sample query for now > anyway. > > I do use the query property to filter which queries get executed. > > I tried turning off auto\_expand\_replica but it seemed to ignore me. > > I manually create the \_percolator index and set it to 2 shards and 0 > replicas but it instantly sets replicas to 1. Can the \_percolator index be > shared like a regular index? And will the engine distribute the percolating > document across shards like the search side of things? > > Can Routing be used with percolate? > > Any other ideas?? > > Thanks in advance! ---
**Author:** ![jschelle\_2](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@jschelle\_2](https://discuss.elastic.co/u/jschelle_2)\ **Post date:** [June 11, 2012, 4:59pm UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019/4 "2012-06-11T16:59:36Z")
Can you explain that please? What part of the percolator design requires the queries to be replicated on every node? I was thinking it could work similar to search: one node could build the index for the document, send it to the other shards, each shard could execute its own local subset of the index and return the matched IDs, the original node could aggregate the responses and return. Would you mind sharing your ideas (whatever part you have formulated so far?)? I'd like to be able to plan accordingly, and might have some time to help with implementation. Just to clarify, you're saying that currently the \_percolator index cannot be sharded like other indices, correct? The percolate documentation[http://www.elasticsearch.org/guide/reference/api/percolate.html](http://www.elasticsearch.org/guide/reference/api/percolate.html)(at least how I read it) makes it sound like it can be sharded by setting index.auto\_expand\_replica to false. If not, what does that setting actually do? Another feature that we might consider in the future is to allow multiple \_percolator indices. (ie. \_percolator1, percolator2, custompercolator, etc) so that the sharding (if possible) can be configured differently for each (non-percolator) index. On Monday, June 11, 2012 10:54:33 AM UTC-4, kimchy wrote: > The design of the \_percolator is such that it needs to exist on all nodes > storage wise. It loads the relevant queries when actual indices are > allocated. I have some ideas on how to improve that, but for now, you can > do the partitioning yourself by having some queries in one "index" (i.e. > /\_percolator/index1) and another in another index (/\_percolator/index2), > and execute the percolate request on both. > > On Fri, Jun 8, 2012 at 1:00 PM, shadow000fire wrote: > > > I'm trying to optimize the performance of my percolators, and hoping > > folks out there have some ideas, as I can't find a lot out there. > > Percolator performance seems to be great until I hit ~100,000 queries, and > > then falls off. Unfortunately I need to be able to support > > 500,000-1,000,000 queries per index. > > > > Queries are already pretty optimized. I'm using a sample query for now > > anyway. > > > > I do use the query property to filter which queries get executed. > > > > I tried turning off auto\_expand\_replica but it seemed to ignore me. > > > > I manually create the \_percolator index and set it to 2 shards and 0 > > replicas but it instantly sets replicas to 1. Can the \_percolator index be > > shared like a regular index? And will the engine distribute the percolating > > document across shards like the search side of things? > > > > Can Routing be used with percolate? > > > > Any other ideas?? > > > > Thanks in advance! ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 13, 2012, 6:13pm UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019/5 "2012-06-13T18:13:10Z")
The \_percolator index needs to be shared among the nodes, and each node loads the relevant data (relevant index is allocated on that node). The design loads all the queries for a relevant index once a shard of it is allocated on that node. Then, when percolating against that index, we can send the percolate request to one of the nodes where a shard of the respective index is allocated, and percolate it there. Because percolation process is very fast, usually, there is no need to distribute it. Note, the \_percolator index just acts as data storage, nothing more. One way to have it distributed is to have percolator queries registered spread across several indices, and do the percolation in parallel against them. The ideas for enhancements are several: 1. Have the ability to execute the percolate request against multiple indices, and parallelize based on that. 2. Allow to specify a parallelism level when percolating, and we can distribute the percolate request based on that, and it will execute only on subsets of the data on each shard. 3. Have a /{index\_name}/\_percolate storage (or something similar) where percolated queries will be stored in a sharded mode, and each percolator request will be distributed across the shards (similar to search). -shay.banon On Mon, Jun 11, 2012 at 6:59 PM, shadow000fire [jason.scheller@gmail.com](mailto:jason.scheller@gmail.com)wrote: > Can you explain that please? What part of the percolator design requires > the queries to be replicated on every node? I was thinking it could work > similar to search: one node could build the index for the document, send it > to the other shards, each shard could execute its own local subset of the > index and return the matched IDs, the original node could aggregate the > responses and return. > > Would you mind sharing your ideas (whatever part you have formulated so > far?)? I'd like to be able to plan accordingly, and might have some time to > help with implementation. > > Just to clarify, you're saying that currently the \_percolator index cannot > be sharded like other indices, correct? The percolate documentation[http://www.elasticsearch.org/guide/reference/api/percolate.html](http://www.elasticsearch.org/guide/reference/api/percolate.html)(at least how I read it) makes it sound like it can be sharded by setting index.auto\_expand\_replica > to false. If not, what does that setting actually do? > > Another feature that we might consider in the future is to allow multiple > \_percolator indices. (ie. \_percolator1, percolator2, custompercolator, etc) > so that the sharding (if possible) can be configured differently for each > (non-percolator) index. > > On Monday, June 11, 2012 10:54:33 AM UTC-4, kimchy wrote: > > > The design of the \_percolator is such that it needs to exist on all nodes > > storage wise. It loads the relevant queries when actual indices are > > allocated. I have some ideas on how to improve that, but for now, you can > > do the partitioning yourself by having some queries in one "index" (i.e. > > /\_percolator/index1) and another in another index (/\_percolator/index2), > > and execute the percolate request on both. > > > > On Fri, Jun 8, 2012 at 1:00 PM, shadow000fire wrote: > > > > > I'm trying to optimize the performance of my percolators, and hoping > > > folks out there have some ideas, as I can't find a lot out there. > > > Percolator performance seems to be great until I hit ~100,000 queries, and > > > then falls off. Unfortunately I need to be able to support > > > 500,000-1,000,000 queries per index. > > > > > > Queries are already pretty optimized. I'm using a sample query for now > > > anyway. > > > > > > I do use the query property to filter which queries get executed. > > > > > > I tried turning off auto\_expand\_replica but it seemed to ignore me. > > > > > > I manually create the \_percolator index and set it to 2 shards and 0 > > > replicas but it instantly sets replicas to 1. Can the \_percolator index be > > > shared like a regular index? And will the engine distribute the percolating > > > document across shards like the search side of things? > > > > > > Can Routing be used with percolate? > > > > > > Any other ideas?? > > > > > > Thanks in advance! ---
**Author:** ![mishu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mishu/32/1942_2.png) [@mishu](https://discuss.elastic.co/u/mishu)\ **Post date:** [July 15, 2013, 12:40pm UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019/6 "2013-07-15T12:40:38Z")
By any chance did any of these improvements get implemented since last year when they were mentioned here? The \_percolator index needs to be shared among the nodes, and each node loads the relevant data (relevant index is allocated on that node). The design loads all the queries for a relevant index once a shard of it is allocated on that node. Then, when percolating against that index, we can send the percolate request to one of the nodes where a shard of the respective index is allocated, and percolate it there. Because percolation process is very fast, usually, there is no need to distribute it. Note, the \_percolator index just acts as data storage, nothing more. One way to have it distributed is to have percolator queries registered spread across several indices, and do the percolation in parallel against them. The ideas for enhancements are several: 1. Have the ability to execute the percolate request against multiple indices, and parallelize based on that. 2. Allow to specify a parallelism level when percolating, and we can distribute the percolate request based on that, and it will execute only on subsets of the data on each shard. 3. Have a /{index\_name}/\_percolate storage (or something similar) where percolated queries will be stored in a sharded mode, and each percolator request will be distributed across the shards (similar to search). -shay.banon ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:26am UTC](https://discuss.elastic.co/t/percolator-performance-ideas/8019/7 "2017-07-06T02:26:28Z")