# Is Bulk Percolation possible? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 8, 2013, 10:25am UTC](https://discuss.elastic.co/t/is-bulk-percolation-possible/11485 "2013-04-08T10:25:38Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [April 8, 2013, 10:25am UTC](https://discuss.elastic.co/t/is-bulk-percolation-possible/11485/1 "2013-04-08T10:25:38Z")
I have a use case were I am receiving a constant stream of documents (on the scale of a thousand document every couple of seconds). and I have a functionality were certain users are interested in only a subset of this and wants to be notified if any new documents of the kind are added. As it stands now I have a bulk being submitted every couple of seconds using [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java) I was wondering if it's possible to percolate in batch as well. i.e. after submitting my batch, the response that I get can include queries and the id's of documents that matched them or a certain field of my choice. I was also wondering about the scale and validity of such a solution, i.e. how many percolators can be present and how does the growth of such number affect perf. in general. Say I have a 1000 users, and each will have 2-3 queries that I need to percolate against. Is this valid? is this too much? if this is not the way to go. What would be the best way to go given that ES is the only place this data lives permanently now. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [April 8, 2013, 6:04pm UTC](https://discuss.elastic.co/t/is-bulk-percolation-possible/11485/2 "2013-04-08T18:04:09Z")
Bulk requests support percolation. You just need to set percolate field on your IndexRequest. As far as performance is concerned, I would suggest testing it. On Monday, April 8, 2013 6:25:38 AM UTC-4, Mo wrote: > I have a use case were I am receiving a constant stream of documents (on > the scale of a thousand document every couple of seconds). and I have a > functionality were certain users are interested in only a subset of this > and wants to be notified if any new documents of the kind are added. > > As it stands now I have a bulk being submitted every couple of seconds > using > [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java) > > I was wondering if it's possible to percolate in batch as well. i.e. after > submitting my batch, the response that I get can include queries and the > id's of documents that matched them or a certain field of my choice. > > I was also wondering about the scale and validity of such a solution, i.e. > how many percolators can be present and how does the growth of such number > affect perf. in general. > > Say I have a 1000 users, and each will have 2-3 queries that I need to > percolate against. Is this valid? is this too much? if this is not the way > to go. What would be the best way to go given that ES is the only place > this data lives permanently now. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [April 9, 2013, 5:23am UTC](https://discuss.elastic.co/t/is-bulk-percolation-possible/11485/3 "2013-04-09T05:23:37Z")
Perfect! Many thanks. On Monday, April 8, 2013 10:04:09 PM UTC+4, Igor Motov wrote: > Bulk requests support percolation. You just need to set percolate field on > your IndexRequest. As far as performance is concerned, I would suggest > testing it. > > On Monday, April 8, 2013 6:25:38 AM UTC-4, Mo wrote: > > > I have a use case were I am receiving a constant stream of documents (on > > the scale of a thousand document every couple of seconds). and I have a > > functionality were certain users are interested in only a subset of this > > and wants to be notified if any new documents of the kind are added. > > > > As it stands now I have a bulk being submitted every couple of seconds > > using > > [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java) > > > > I was wondering if it's possible to percolate in batch as well. i.e. > > after submitting my batch, the response that I get can include queries and > > the id's of documents that matched them or a certain field of my choice. > > > > I was also wondering about the scale and validity of such a solution, > > i.e. how many percolators can be present and how does the growth of such > > number affect perf. in general. > > > > Say I have a 1000 users, and each will have 2-3 queries that I need to > > percolate against. Is this valid? is this too much? if this is not the way > > to go. What would be the best way to go given that ES is the only place > > this data lives permanently now. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:42am UTC](https://discuss.elastic.co/t/is-bulk-percolation-possible/11485/4 "2017-07-06T02:42:08Z")