# Percolation while indexing to rolling indexes (logs) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 28, 2013, 12:44pm UTC](https://discuss.elastic.co/t/percolation-while-indexing-to-rolling-indexes-logs/10524 "2013-01-28T12:44:22Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [January 28, 2013, 12:44pm UTC](https://discuss.elastic.co/t/percolation-while-indexing-to-rolling-indexes-logs/10524/1 "2013-01-28T12:44:22Z")
Here's an interesting problem. Our system is built on ElasticSearch, and is using the rolling indexing technique - that is, we have an index per time period. This approach is also know in this forum as "indexing logs". We use the percolation feature of ElasticSearch, and are interested in percolating while indexing. The problem with doing that in this setup is percolator queries are registered against a specific index, and that's not necessarily the index we are indexing to. The immediate solution is to register all queries to all indexes we have in the system, but that's just ridiculous, as queries are updated and removed over time. What we want to have is a pseudo index we can associate all queries with, and load them from the percolator if an appropriate flag was set in the percolation / indexing request. A wildcard-based solution could work as well. I'm looking at this line: > indexName there should be set to that psuedo-index all queries are registered against. There may be better ways to do that - we would appreciate feedback from ES devs and community. If this is the best way to go, we would be happy to provide a pull request adding that feature. Itamar. ---
**Author:** ![Loic\_Bertron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loic_bertron/32/1794_2.png) [@Loic\_Bertron](https://discuss.elastic.co/u/Loic_Bertron)\ **Post date:** [January 29, 2013, 3:48pm UTC](https://discuss.elastic.co/t/percolation-while-indexing-to-rolling-indexes-logs/10524/2 "2013-01-29T15:48:21Z")
Hey, Since every index have the same mapping, I would suggest to create an empty index and to register your percolator queries against this index. Instead of adding percolate parameter to the index process, you could simply query your percolator index to check if your document match one or many queries > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. But anyways, there is no way to percolate and index in the same time with rolling indexes. Le lundi 28 janvier 2013 07:44:22 UTC-5, Itamar Syn-Hershko a écrit : > Here's an interesting problem. > > Our system is built on Elasticsearch, and is using the rolling indexing > technique - that is, we have an index per time period. This approach is > also know in this forum as "indexing logs". > > We use the percolation feature of Elasticsearch, and are interested in > percolating while indexing. The problem with doing that in this setup is > percolator queries are registered against a specific index, and that's not > necessarily the index we are indexing to. > > The immediate solution is to register all queries to all indexes we have > in the system, but that's just ridiculous, as queries are updated and > removed over time. > > What we want to have is a pseudo index we can associate all queries with, > and load them from the percolator if an appropriate flag was set in the > percolation / indexing request. A wildcard-based solution could work as > well. > > I'm looking at this line: > [https://github.com/elasticsearch/elasticsearch/blob/ea9a4d70cf7140d6cf5c3c12e59fca0718164f4a/src/main/java/org/elasticsearch/index/percolator/PercolatorService.java#L126](https://github.com/elasticsearch/elasticsearch/blob/ea9a4d70cf7140d6cf5c3c12e59fca0718164f4a/src/main/java/org/elasticsearch/index/percolator/PercolatorService.java#L126) > > indexName there should be set to that psuedo-index all queries are > registered against. > > There may be better ways to do that - we would appreciate feedback from ES > devs and community. If this is the best way to go, we would be happy to > provide a pull request adding that feature. > > Itamar. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [January 29, 2013, 4:10pm UTC](https://discuss.elastic.co/t/percolation-while-indexing-to-rolling-indexes-logs/10524/3 "2013-01-29T16:10:13Z")
This is what we do now, but its twice the work. Hence my question about the best way to make this possible while indexing, with rolling indexes as well. On Tue, Jan 29, 2013 at 5:48 PM, Loïc Bertron [loic.bertron@gmail.com](mailto:loic.bertron@gmail.com)wrote: > Hey, > > Since every index have the same mapping, I would suggest to create an > empty index and to register your percolator queries against this index. > Instead of adding percolate parameter to the index process, you could > simply query your percolator index to check if your document match one or > many queries > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/percolate.html) > > But anyways, there is no way to percolate and index in the same time with > rolling indexes. > > Le lundi 28 janvier 2013 07:44:22 UTC-5, Itamar Syn-Hershko a écrit : > > > Here's an interesting problem. > > > > Our system is built on Elasticsearch, and is using the rolling indexing > > technique - that is, we have an index per time period. This approach is > > also know in this forum as "indexing logs". > > > > We use the percolation feature of Elasticsearch, and are interested in > > percolating while indexing. The problem with doing that in this setup is > > percolator queries are registered against a specific index, and that's not > > necessarily the index we are indexing to. > > > > The immediate solution is to register all queries to all indexes we have > > in the system, but that's just ridiculous, as queries are updated and > > removed over time. > > > > What we want to have is a pseudo index we can associate all queries with, > > and load them from the percolator if an appropriate flag was set in the > > percolation / indexing request. A wildcard-based solution could work as > > well. > > > > I'm looking at this line: [https://github.com/](https://github.com/)\*\* > > elasticsearch/elasticsearch/\*\*blob/ **ea9a4d70cf7140d6cf5c3c12e59fca** > > 0718164f4a/src/main/java/org/ **elasticsearch/index/** > > percolator/PercolatorService.\*\*java#L126[https://github.com/elasticsearch/elasticsearch/blob/ea9a4d70cf7140d6cf5c3c12e59fca0718164f4a/src/main/java/org/elasticsearch/index/percolator/PercolatorService.java#L126](https://github.com/elasticsearch/elasticsearch/blob/ea9a4d70cf7140d6cf5c3c12e59fca0718164f4a/src/main/java/org/elasticsearch/index/percolator/PercolatorService.java#L126) > > > > indexName there should be set to that psuedo-index all queries are > > registered against. > > > > There may be better ways to do that - we would appreciate feedback from > > ES devs and community. If this is the best way to go, we would be happy to > > provide a pull request adding that feature. > > > > Itamar. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [February 3, 2013, 1:08pm UTC](https://discuss.elastic.co/t/percolation-while-indexing-to-rolling-indexes-logs/10524/4 "2013-02-03T13:08:35Z")
Hi again, I ended up with what I believe to be a simple and elegant solution, although I might have gotten some of the naming wrong. Here's a pull request: > > > By having an index called \_global with mapping that match the mapping of the rol…ling indices in your system, and registering all percolator queries against this index, you can percolate and index with percolation and get results for all registered queries without worrying about registering all query on all indices in your system. > > See discussion here: http://elasticsearch-users.115913.n3.nabble.com/Percolation-while-indexing-to-rolling-indexes-logs-td4028858.html What I did was to define a new "system index", which I named \_global, and registering queries against it will ensure they participate in all percolation operations within the cluster along with queries registered against the original query. To make this work you need to define an index named "\_global", and to give it some mapping. Since this is intended to be used in clusters employing the "rolling indices" pattern, it is safe to assume the mapping of \_global will match the mapping of any other index in the system, and this is what did. Comments welcome. Itamar. On Tue, Jan 29, 2013 at 6:10 PM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > This is what we do now, but its twice the work. Hence my question about > the best way to make this possible while indexing, with rolling indexes as > well. > > On Tue, Jan 29, 2013 at 5:48 PM, Loïc Bertron [loic.bertron@gmail.com](mailto:loic.bertron@gmail.com)wrote: > > > Hey, > > > > Since every index have the same mapping, I would suggest to create an > > empty index and to register your percolator queries against this index. > > Instead of adding percolate parameter to the index process, you could > > simply query your percolator index to check if your document match one or > > many queries > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/percolate.html) > > > > But anyways, there is no way to percolate and index in the same time with > > rolling indexes. > > > > Le lundi 28 janvier 2013 07:44:22 UTC-5, Itamar Syn-Hershko a écrit : > > > > > Here's an interesting problem. > > > > > > Our system is built on Elasticsearch, and is using the rolling indexing > > > technique - that is, we have an index per time period. This approach is > > > also know in this forum as "indexing logs". > > > > > > We use the percolation feature of Elasticsearch, and are interested in > > > percolating while indexing. The problem with doing that in this setup is > > > percolator queries are registered against a specific index, and that's not > > > necessarily the index we are indexing to. > > > > > > The immediate solution is to register all queries to all indexes we have > > > in the system, but that's just ridiculous, as queries are updated and > > > removed over time. > > > > > > What we want to have is a pseudo index we can associate all queries > > > with, and load them from the percolator if an appropriate flag was set in > > > the percolation / indexing request. A wildcard-based solution could work as > > > well. > > > > > > I'm looking at this line: [https://github.com/](https://github.com/)\*\* > > > elasticsearch/elasticsearch/\*\*blob/ **ea9a4d70cf7140d6cf5c3c12e59fca** > > > 0718164f4a/src/main/java/org/ **elasticsearch/index/** > > > percolator/PercolatorService.\*\*java#L126[https://github.com/elasticsearch/elasticsearch/blob/ea9a4d70cf7140d6cf5c3c12e59fca0718164f4a/src/main/java/org/elasticsearch/index/percolator/PercolatorService.java#L126](https://github.com/elasticsearch/elasticsearch/blob/ea9a4d70cf7140d6cf5c3c12e59fca0718164f4a/src/main/java/org/elasticsearch/index/percolator/PercolatorService.java#L126) > > > > > > indexName there should be set to that psuedo-index all queries are > > > registered against. > > > > > > There may be better ways to do that - we would appreciate feedback from > > > ES devs and community. If this is the best way to go, we would be happy to > > > provide a pull request adding that feature. > > > > > > Itamar. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:53am UTC](https://discuss.elastic.co/t/percolation-while-indexing-to-rolling-indexes-logs/10524/5 "2017-07-06T02:53:17Z")