# Slow Percolator Indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [March 3, 2014, 9:47pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121 "2014-03-03T21:47:11Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![James\_Bathgate\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_bathgate_2/32/1758_2.png) [@James\_Bathgate\_2](https://discuss.elastic.co/u/James_Bathgate_2)\ **Post date:** [March 3, 2014, 9:47pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/1 "2014-03-03T21:47:11Z")
I'm having issues with indexing percolator queries taking a long time to insert. I created a sample test case: > As you can see it's taking 1.5s to insert a single percolator. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bb7d4a88-e194-4806-9583-d2dcc197fcfc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bb7d4a88-e194-4806-9583-d2dcc197fcfc%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 3, 2014, 10:00pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/2 "2014-03-03T22:00:53Z")
Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java 1.7\_u51 and here are my results: Insert Percolator: {"\_index":"merchandising","\_type":".percolator","\_id":"1","\_version":1,"created":true} real 0m0.103s user 0m0.015s sys 0m0.000s Is it possible that your index was just busy? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bbb5dd8f-281f-48bf-90d5-002d9c8b8b6e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bbb5dd8f-281f-48bf-90d5-002d9c8b8b6e%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![James\_Bathgate\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_bathgate_2/32/1758_2.png) [@James\_Bathgate\_2](https://discuss.elastic.co/u/James_Bathgate_2)\ **Post date:** [March 3, 2014, 10:14pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/3 "2014-03-03T22:14:34Z")
I don't think so. This is on my local dev with nothing else running on the index. On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java 1.7\_u51 and > here are my results: > > Insert Percolator: > > {"\_index":"merchandising","\_type":".percolator","\_id":"1","\_version":1,"created":true} > real 0m0.103s > user 0m0.015s > sys 0m0.000s > > Is it possible that your index was just busy? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b7fa0a36-36e4-4276-9ce7-b4af10c103ce%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7fa0a36-36e4-4276-9ce7-b4af10c103ce%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![James\_Bathgate\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_bathgate_2/32/1758_2.png) [@James\_Bathgate\_2](https://discuss.elastic.co/u/James_Bathgate_2)\ **Post date:** [March 3, 2014, 11:25pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/4 "2014-03-03T23:25:01Z")
What are the likely bottlenecks on indexing percolators? This is on a Vagrant virtual machine with 2 GBs of RAM with 1GB for ES. I'm wondering if this won't be a problem on an EC2 instance. On Monday, March 3, 2014 2:14:34 PM UTC-8, James Bathgate wrote: > I don't think so. This is on my local dev with nothing else running on the > index. > > On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > > > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java 1.7\_u51 and > > here are my results: > > > > Insert Percolator: > > > > {"\_index":"merchandising","\_type":".percolator","\_id":"1","\_version":1,"created":true} > > real 0m0.103s > > user 0m0.015s > > sys 0m0.000s > > > > Is it possible that your index was just busy? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [March 3, 2014, 11:40pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/5 "2014-03-03T23:40:05Z")
On top of just indexing a document, the top level 'query' field part gets parsed into a internal Lucene query. However I don't see why this should take a long time. Some questions: Are you running low on jvm memory? How long does it take to index a regular document? Can you run the hot threads api while indexing a percolator document? Martijn On 4 March 2014 00:25, James Bathgate [james@b7interactive.com](mailto:james@b7interactive.com) wrote: > What are the likely bottlenecks on indexing percolators? This is on a > Vagrant virtual machine with 2 GBs of RAM with 1GB for ES. I'm wondering if > this won't be a problem on an EC2 instance. > > On Monday, March 3, 2014 2:14:34 PM UTC-8, James Bathgate wrote: > > > I don't think so. This is on my local dev with nothing else running on > > the index. > > > > On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > > > > > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java 1.7\_u51 > > > and here are my results: > > > > > > Insert Percolator: > > > {"\_index":"merchandising","\_type":".percolator","\_id":"1", > > > "\_version":1,"created":true} > > > real 0m0.103s > > > user 0m0.015s > > > sys 0m0.000s > > > > > > Is it possible that your index was just busy? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tx46ABEpAS94-EO7WGui05Y1RSpx%3DaNGMnHuTHgHf4orA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tx46ABEpAS94-EO7WGui05Y1RSpx%3DaNGMnHuTHgHf4orA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![James\_Bathgate\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_bathgate_2/32/1758_2.png) [@James\_Bathgate\_2](https://discuss.elastic.co/u/James_Bathgate_2)\ **Post date:** [March 4, 2014, 12:33am UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/6 "2014-03-04T00:33:19Z")
Martijn, 1. Not running low at all. 2. A regular document takes ~75ms. 3. [gist:9337810 · GitHub](https://gist.github.com/julesbravo/9337810) It looks like it's definitely CPU bound. On Monday, March 3, 2014 3:40:05 PM UTC-8, Martijn v Groningen wrote: > On top of just indexing a document, the top level 'query' field part gets > parsed into a internal Lucene query. However I don't see why this should > take a long time. > > Some questions: > Are you running low on jvm memory? > How long does it take to index a regular document? > Can you run the hot threads api while indexing a percolator document? > > Martijn > > On 4 March 2014 00:25, James Bathgate \<[ja...@b7interactive.com](mailto:ja...@b7interactive.com)\ > > > wrote: > > > What are the likely bottlenecks on indexing percolators? This is on a > > Vagrant virtual machine with 2 GBs of RAM with 1GB for ES. I'm wondering if > > this won't be a problem on an EC2 instance. > > > > On Monday, March 3, 2014 2:14:34 PM UTC-8, James Bathgate wrote: > > > > > I don't think so. This is on my local dev with nothing else running on > > > the index. > > > > > > On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > > > > > > > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java 1.7\_u51 > > > > and here are my results: > > > > > > > > Insert Percolator: > > > > {"\_index":"merchandising","\_type":".percolator","\_id":"1", > > > > "\_version":1,"created":true} > > > > real 0m0.103s > > > > user 0m0.015s > > > > sys 0m0.000s > > > > > > > > Is it possible that your index was just busy? > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40googlegroups.com) > > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Met vriendelijke groet, > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/74560785-adc5-4fe1-885c-71179b2b29f1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/74560785-adc5-4fe1-885c-71179b2b29f1%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [March 4, 2014, 9:05am UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/7 "2014-03-04T09:05:13Z")
I see that you a lot of time is spend on just measuring how memory the query takes in memory and not parsing the query. I think this slowness might be jvm version dependent, what jvm version are you using? On 4 March 2014 01:33, James Bathgate [james@b7interactive.com](mailto:james@b7interactive.com) wrote: > Martijn, > > 1. Not running low at all. > 2. A regular document takes ~75ms. > 3. [gist:9337810 · GitHub](https://gist.github.com/julesbravo/9337810) > > It looks like it's definitely CPU bound. > > On Monday, March 3, 2014 3:40:05 PM UTC-8, Martijn v Groningen wrote: > > > On top of just indexing a document, the top level 'query' field part gets > > parsed into a internal Lucene query. However I don't see why this should > > take a long time. > > > > Some questions: > > Are you running low on jvm memory? > > How long does it take to index a regular document? > > Can you run the hot threads api while indexing a percolator document? > > > > Martijn > > > > On 4 March 2014 00:25, James Bathgate [ja...@b7interactive.com](mailto:ja...@b7interactive.com) wrote: > > > > > What are the likely bottlenecks on indexing percolators? This is on a > > > Vagrant virtual machine with 2 GBs of RAM with 1GB for ES. I'm wondering if > > > this won't be a problem on an EC2 instance. > > > > > > On Monday, March 3, 2014 2:14:34 PM UTC-8, James Bathgate wrote: > > > > > > > I don't think so. This is on my local dev with nothing else running on > > > > the index. > > > > > > > > On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > > > > > > > > > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java 1.7\_u51 > > > > > and here are my results: > > > > > > > > > > Insert Percolator: > > > > > {"\_index":"merchandising","\_type":".percolator","_id":"1","_ > > > > > version":1,"created":true} > > > > > real 0m0.103s > > > > > user 0m0.015s > > > > > sys 0m0.000s > > > > > > > > > > Is it possible that your index was just busy? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294% > > > > [40googlegroups.com](http://40googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BA76TwgTXRmo4tCsNvbag5%3DUiMj3C4uBSfD5nJiMwnAHhEiYA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BA76TwgTXRmo4tCsNvbag5%3DUiMj3C4uBSfD5nJiMwnAHhEiYA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![James\_Bathgate\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_bathgate_2/32/1758_2.png) [@James\_Bathgate\_2](https://discuss.elastic.co/u/James_Bathgate_2)\ **Post date:** [March 4, 2014, 3:52pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/8 "2014-03-04T15:52:39Z")
Martijn, I'm using Oracle Java 7u45 On Tuesday, March 4, 2014 1:05:13 AM UTC-8, Martijn v Groningen wrote: > I see that you a lot of time is spend on just measuring how memory the > query takes in memory and not parsing the query. I think this slowness > might be jvm version dependent, what jvm version are you using? > > On 4 March 2014 01:33, James Bathgate \<[ja...@b7interactive.com](mailto:ja...@b7interactive.com)\ > > > wrote: > > > Martijn, > > > > 1. Not running low at all. > > 2. A regular document takes ~75ms. > > 3. [gist:9337810 · GitHub](https://gist.github.com/julesbravo/9337810) > > > > It looks like it's definitely CPU bound. > > > > On Monday, March 3, 2014 3:40:05 PM UTC-8, Martijn v Groningen wrote: > > > > > On top of just indexing a document, the top level 'query' field part > > > gets parsed into a internal Lucene query. However I don't see why this > > > should take a long time. > > > > > > Some questions: > > > Are you running low on jvm memory? > > > How long does it take to index a regular document? > > > Can you run the hot threads api while indexing a percolator document? > > > > > > Martijn > > > > > > On 4 March 2014 00:25, James Bathgate [ja...@b7interactive.com](mailto:ja...@b7interactive.com) wrote: > > > > > > > What are the likely bottlenecks on indexing percolators? This is on a > > > > Vagrant virtual machine with 2 GBs of RAM with 1GB for ES. I'm wondering if > > > > this won't be a problem on an EC2 instance. > > > > > > > > On Monday, March 3, 2014 2:14:34 PM UTC-8, James Bathgate wrote: > > > > > > > > > I don't think so. This is on my local dev with nothing else running on > > > > > the index. > > > > > > > > > > On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > > > > > > > > > > > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java 1.7\_u51 > > > > > > and here are my results: > > > > > > > > > > > > Insert Percolator: > > > > > > {"\_index":"merchandising","\_type":".percolator","_id":"1","_ > > > > > > version":1,"created":true} > > > > > > real 0m0.103s > > > > > > user 0m0.015s > > > > > > sys 0m0.000s > > > > > > > > > > > > Is it possible that your index was just busy? > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294% > > > > > [40googlegroups.com](http://40googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > Met vriendelijke groet, > > > > > > Martijn van Groningen > > -- > Met vriendelijke groet, > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/606b8eff-5085-4970-b10e-17590c19c12b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/606b8eff-5085-4970-b10e-17590c19c12b%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [March 4, 2014, 9:45pm UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/9 "2014-03-04T21:45:29Z")
Ok, I thought may be an old jvm version was causing this, but this one is pretty recent. I took a better look at indexing percolator queries and there is indeed a substantial difference in execution time comparing to indexing a regular document. When I disabled the size calculation (in the code) for percolator queries the execution between indexing a regular document and an percolator document is more or less the same. I opened an issue for this: > > > The percolator keeps track of the total amount memory being spent on all the que…ries that are in memory, this statistic is then exposed in the node stats, indices stats and cluster stats apis. Each time a percolator query is registered or deleted its size in memory is calculated and that is added or subtracted from the total size in bytes all percolator queries take in memory. The percolator uses Lucene's \`RamUsageEstimator\` which takes a substantial chunk of the total time being spend on registering a percolator query. > > I think the total amount of memory spent on queries is a valuable statistic, but it shouldn't add a substantial overhead in registering a percolator query. Therefor I think we should compute the memory spent on percolator queries on the fly in the stats api, but only when the percolator stats are specifically requested. Downside would be that it may take a while for the \`memory spent on percolator queries\` statistic to estimate. > > Based on: https://groups.google.com/forum/#!topic/elasticsearch/6bgUvea98Uw On 4 March 2014 16:52, James Bathgate [james@b7interactive.com](mailto:james@b7interactive.com) wrote: > Martijn, > > I'm using Oracle Java 7u45 > > On Tuesday, March 4, 2014 1:05:13 AM UTC-8, Martijn v Groningen wrote: > > > I see that you a lot of time is spend on just measuring how memory the > > query takes in memory and not parsing the query. I think this slowness > > might be jvm version dependent, what jvm version are you using? > > > > On 4 March 2014 01:33, James Bathgate [ja...@b7interactive.com](mailto:ja...@b7interactive.com) wrote: > > > > > Martijn, > > > > > > 1. Not running low at all. > > > 2. A regular document takes ~75ms. > > > 3. [gist:9337810 · GitHub](https://gist.github.com/julesbravo/9337810) > > > > > > It looks like it's definitely CPU bound. > > > > > > On Monday, March 3, 2014 3:40:05 PM UTC-8, Martijn v Groningen wrote: > > > > > > > On top of just indexing a document, the top level 'query' field part > > > > gets parsed into a internal Lucene query. However I don't see why this > > > > should take a long time. > > > > > > > > Some questions: > > > > Are you running low on jvm memory? > > > > How long does it take to index a regular document? > > > > Can you run the hot threads api while indexing a percolator document? > > > > > > > > Martijn > > > > > > > > On 4 March 2014 00:25, James Bathgate [ja...@b7interactive.com](mailto:ja...@b7interactive.com) wrote: > > > > > > > > > What are the likely bottlenecks on indexing percolators? This is on a > > > > > Vagrant virtual machine with 2 GBs of RAM with 1GB for ES. I'm wondering if > > > > > this won't be a problem on an EC2 instance. > > > > > > > > > > On Monday, March 3, 2014 2:14:34 PM UTC-8, James Bathgate wrote: > > > > > > > > > > > I don't think so. This is on my local dev with nothing else running > > > > > > on the index. > > > > > > > > > > > > On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > > > > > > > > > > > > > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java > > > > > > > 1.7\_u51 and here are my results: > > > > > > > > > > > > > > Insert Percolator: > > > > > > > {"\_index":"merchandising","\_type":".percolator","_id":"1","_ > > > > > > > version":1,"created":true} > > > > > > > real 0m0.103s > > > > > > > user 0m0.015s > > > > > > > sys 0m0.000s > > > > > > > > > > > > > > Is it possible that your index was just busy? > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40goo > > > > > > [glegroups.com](http://glegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > Met vriendelijke groet, > > > > > > > > Martijn van Groningen > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tz2rkid%3D0a1Ar5oS2\_R\_yo%2BQG32XwYdgsWLdiOxws\_O\_Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BA76Tz2rkid%3D0a1Ar5oS2_R_yo%2BQG32XwYdgsWLdiOxws_O_Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![James\_Bathgate\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_bathgate_2/32/1758_2.png) [@James\_Bathgate\_2](https://discuss.elastic.co/u/James_Bathgate_2)\ **Post date:** [March 5, 2014, 12:02am UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/10 "2014-03-05T00:02:05Z")
Thanks. On Tuesday, March 4, 2014 1:45:29 PM UTC-8, Martijn v Groningen wrote: > Ok, I thought may be an old jvm version was causing this, but this one is > pretty recent. > > I took a better look at indexing percolator queries and there is indeed a > substantial difference in execution time comparing to indexing a regular > document. When I disabled the size calculation (in the code) for percolator > queries the execution between indexing a regular document and an percolator > document is more or less the same. > > I opened an issue for this: > [The size estimation for percolator queries takes longer than it should · Issue #5339 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/5339) > > On 4 March 2014 16:52, James Bathgate \<[ja...@b7interactive.com](mailto:ja...@b7interactive.com)\ > > > wrote: > > > Martijn, > > > > I'm using Oracle Java 7u45 > > > > On Tuesday, March 4, 2014 1:05:13 AM UTC-8, Martijn v Groningen wrote: > > > > > I see that you a lot of time is spend on just measuring how memory the > > > query takes in memory and not parsing the query. I think this slowness > > > might be jvm version dependent, what jvm version are you using? > > > > > > On 4 March 2014 01:33, James Bathgate [ja...@b7interactive.com](mailto:ja...@b7interactive.com) wrote: > > > > > > > Martijn, > > > > > > > > 1. Not running low at all. > > > > 2. A regular document takes ~75ms. > > > > 3. [gist:9337810 · GitHub](https://gist.github.com/julesbravo/9337810) > > > > > > > > It looks like it's definitely CPU bound. > > > > > > > > On Monday, March 3, 2014 3:40:05 PM UTC-8, Martijn v Groningen wrote: > > > > > > > > > On top of just indexing a document, the top level 'query' field part > > > > > gets parsed into a internal Lucene query. However I don't see why this > > > > > should take a long time. > > > > > > > > > > Some questions: > > > > > Are you running low on jvm memory? > > > > > How long does it take to index a regular document? > > > > > Can you run the hot threads api while indexing a percolator document? > > > > > > > > > > Martijn > > > > > > > > > > On 4 March 2014 00:25, James Bathgate [ja...@b7interactive.com](mailto:ja...@b7interactive.com) wrote: > > > > > > > > > > > What are the likely bottlenecks on indexing percolators? This is on a > > > > > > Vagrant virtual machine with 2 GBs of RAM with 1GB for ES. I'm wondering if > > > > > > this won't be a problem on an EC2 instance. > > > > > > > > > > > > On Monday, March 3, 2014 2:14:34 PM UTC-8, James Bathgate wrote: > > > > > > > > > > > > > I don't think so. This is on my local dev with nothing else running > > > > > > > on the index. > > > > > > > > > > > > > > On Monday, March 3, 2014 2:00:53 PM UTC-8, Binh Ly wrote: > > > > > > > > > > > > > > > Hmmmm not sure, I just tried this on ES 1.0.1 and Oracle Java > > > > > > > > 1.7\_u51 and here are my results: > > > > > > > > > > > > > > > > Insert Percolator: > > > > > > > > {"\_index":"merchandising","\_type":".percolator","_id":"1","_ > > > > > > > > version":1,"created":true} > > > > > > > > real 0m0.103s > > > > > > > > user 0m0.015s > > > > > > > > sys 0m0.000s > > > > > > > > > > > > > > > > Is it possible that your index was just busy? > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > > msgid/elasticsearch/5dc16213-ba8e-4d68-9518-0882bd26e294%40goo > > > > > > > [glegroups.com](http://glegroups.com). > > > > > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > -- > > > > > Met vriendelijke groet, > > > > > > > > > > Martijn van Groningen > > > > > > -- > > > Met vriendelijke groet, > > > > > > Martijn van Groningen > > -- > Met vriendelijke groet, > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fe29c8e-72e6-440a-b292-f1fbdba2df03%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fe29c8e-72e6-440a-b292-f1fbdba2df03%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:45am UTC](https://discuss.elastic.co/t/slow-percolator-indexing/16121/11 "2017-07-06T01:45:42Z")