# Search performance **URL:** \ **Category:** Elasticsearch\ **Created:** [August 24, 2010, 7:06pm UTC](https://discuss.elastic.co/t/search-performance/3255 "2010-08-24T19:06:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [August 24, 2010, 7:06pm UTC](https://discuss.elastic.co/t/search-performance/3255/1 "2010-08-24T19:06:27Z")
Just wanted to post how ecstatic I am with search performance. My h/w is 24 CPU cores w/ 48GB or ram. I have 45 indexes total with the largest getting a couple of shards, for a total of 75 shards. I have 20 million documents with an index size of ~28GB. I am able to pump through ~450 real world queries per second. The queries are very large boolean queries, some with wildcards and prefix queries and represent 10+ years of legacy queries that are far from optimal in most cases. These are not the standard simple test queries. Also, these are queries that have made it past our caching layer, implying that there is a lot of variation in the requests. The obvious bottleneck ends up being CPU saturation. Disk load looks negligible. If I start a flood of documents indexing, things slow down a little, but not much. ES consumes ~10GB of JVM heap in this setup. I'm assuming that the system is using the rest of the RAM for disk cache. Thanks for the great work! Paul ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [August 25, 2010, 6:24am UTC](https://discuss.elastic.co/t/search-performance/3255/2 "2010-08-25T06:24:57Z")
And, of course, I was still bounded client side 🙂 Was able to ramp up to 1000 queries/sec with no warm up. I believe that my indexes are pegged in system cache. Really awesome. On Aug 24, 1:06 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > Just wanted to post how ecstatic I am with search performance. > > My h/w is 24 CPU cores w/ 48GB or ram. I have 45 indexes total with > the largest getting a couple of shards, for a total of 75 shards. I > have 20 million documents with an index size of ~28GB. > > I am able to pump through ~450 real world queries per second. The > queries are very large boolean queries, some with wildcards and prefix > queries and represent 10+ years of legacy queries that are far from > optimal in most cases. These are not the standard simple test queries. > Also, these are queries that have made it past our caching layer, > implying that there is a lot of variation in the requests. > > The obvious bottleneck ends up being CPU saturation. Disk load looks > negligible. > > If I start a flood of documents indexing, things slow down a little, > but not much. > > ES consumes ~10GB of JVM heap in this setup. > > I'm assuming that the system is using the rest of the RAM for disk > cache. > > Thanks for the great work! > > Paul ---
**Author:** ![Stephen\_Day\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephen_day_2/32/3323_2.png) [@Stephen\_Day\_2](https://discuss.elastic.co/u/Stephen_Day_2)\ **Post date:** [August 25, 2010, 3:06pm UTC](https://discuss.elastic.co/t/search-performance/3255/3 "2010-08-25T15:06:32Z")
Is this a single machine (24 cores, 48 GB RAM) or are you clustered across several machines (3 machines 16GB RAM, 8 cores each)? On Tue, Aug 24, 2010 at 11:24 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > And, of course, I was still bounded client side 🙂 > > Was able to ramp up to 1000 queries/sec with no warm up. I believe > that my indexes are pegged in system cache. > > Really awesome. > > On Aug 24, 1:06 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > Just wanted to post how ecstatic I am with search performance. > > > > My h/w is 24 CPU cores w/ 48GB or ram. I have 45 indexes total with > > the largest getting a couple of shards, for a total of 75 shards. I > > have 20 million documents with an index size of ~28GB. > > > > I am able to pump through ~450 real world queries per second. The > > queries are very large boolean queries, some with wildcards and prefix > > queries and represent 10+ years of legacy queries that are far from > > optimal in most cases. These are not the standard simple test queries. > > Also, these are queries that have made it past our caching layer, > > implying that there is a lot of variation in the requests. > > > > The obvious bottleneck ends up being CPU saturation. Disk load looks > > negligible. > > > > If I start a flood of documents indexing, things slow down a little, > > but not much. > > > > ES consumes ~10GB of JVM heap in this setup. > > > > I'm assuming that the system is using the rest of the RAM for disk > > cache. > > > > Thanks for the great work! > > > > Paul ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [August 25, 2010, 5:43pm UTC](https://discuss.elastic.co/t/search-performance/3255/4 "2010-08-25T17:43:08Z")
Hi Stephen, This is a single 24 core, 48GB of RAM machine. And I have further revised numbers of over 1500 queries per second, with only a handful (less than 0.1%) running at over 2 seconds and an average running at 34ms. To add a comparison point, our current legacy search system (which I would name if I could) can pump through ~80 of the same queries against the same data set per second on semi-similar h/w. The really cool thing is that there just does not seem to be a bottleneck in my config other than CPU. I was really expecting to have to tweak things to work around memory or disk bottlnecks, but that has not been the case. Thanks, Paul On Aug 25, 9:06 am, Stephen Day [stevv...@gmail.com](mailto:stevv...@gmail.com) wrote: > Is this a single machine (24 cores, 48 GB RAM) or are you clustered > across several machines (3 machines 16GB RAM, 8 cores each)? > > On Tue, Aug 24, 2010 at 11:24 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > And, of course, I was still bounded client side 🙂 > > > Was able to ramp up to 1000 queries/sec with no warm up. I believe > > that my indexes are pegged in system cache. > > > Really awesome. > > > On Aug 24, 1:06 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > Just wanted to post how ecstatic I am with search performance. > > > > My h/w is 24 CPU cores w/ 48GB or ram. I have 45 indexes total with > > > the largest getting a couple of shards, for a total of 75 shards. I > > > have 20 million documents with an index size of ~28GB. > > > > I am able to pump through ~450 real world queries per second. The > > > queries are very large boolean queries, some with wildcards and prefix > > > queries and represent 10+ years of legacy queries that are far from > > > optimal in most cases. These are not the standard simple test queries. > > > Also, these are queries that have made it past our caching layer, > > > implying that there is a lot of variation in the requests. > > > > The obvious bottleneck ends up being CPU saturation. Disk load looks > > > negligible. > > > > If I start a flood of documents indexing, things slow down a little, > > > but not much. > > > > ES consumes ~10GB of JVM heap in this setup. > > > > I'm assuming that the system is using the rest of the RAM for disk > > > cache. > > > > Thanks for the great work! > > > > Paul ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [August 25, 2010, 7:52pm UTC](https://discuss.elastic.co/t/search-performance/3255/5 "2010-08-25T19:52:41Z")
OK, major DOH! on my part. Current numbers are ~500 queries per sec, averaging ~110ms per w/ a fraction of a percent running in over 2 seconds. My initial numbers of ~500 queries per sec where run from python scripts on windows boxes. I moved these scripts over to more powerful linux boxes to try to eliminate any client side bottleneck. Unfortunately, the time.clock() behaves dramatically differently on windows vs linux and the larger numbers I posted where a fantasy. Still really happy with current results, though 🙂 Now to distribute. On Aug 25, 11:43 am, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > Hi Stephen, > This is a single 24 core, 48GB of RAM machine. And I have further > revised numbers of over 1500 queries per second, with only a handful > (less than 0.1%) running at over 2 seconds and an average running at > 34ms. > > To add a comparison point, our current legacy search system (which I > would name if I could) can pump through ~80 of the same queries > against the same data set per second on semi-similar h/w. > > The really cool thing is that there just does not seem to be a > bottleneck in my config other than CPU. I was really expecting to have > to tweak things to work around memory or disk bottlnecks, but that has > not been the case. > > Thanks, > Paul > > On Aug 25, 9:06 am, Stephen Day [stevv...@gmail.com](mailto:stevv...@gmail.com) wrote: > > > Is this a single machine (24 cores, 48 GB RAM) or are you clustered > > across several machines (3 machines 16GB RAM, 8 cores each)? > > > On Tue, Aug 24, 2010 at 11:24 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > And, of course, I was still bounded client side 🙂 > > > > Was able to ramp up to 1000 queries/sec with no warm up. I believe > > > that my indexes are pegged in system cache. > > > > Really awesome. > > > > On Aug 24, 1:06 pm, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > > > Just wanted to post how ecstatic I am with search performance. > > > > > My h/w is 24 CPU cores w/ 48GB or ram. I have 45 indexes total with > > > > the largest getting a couple of shards, for a total of 75 shards. I > > > > have 20 million documents with an index size of ~28GB. > > > > > I am able to pump through ~450 real world queries per second. The > > > > queries are very large boolean queries, some with wildcards and prefix > > > > queries and represent 10+ years of legacy queries that are far from > > > > optimal in most cases. These are not the standard simple test queries. > > > > Also, these are queries that have made it past our caching layer, > > > > implying that there is a lot of variation in the requests. > > > > > The obvious bottleneck ends up being CPU saturation. Disk load looks > > > > negligible. > > > > > If I start a flood of documents indexing, things slow down a little, > > > > but not much. > > > > > ES consumes ~10GB of JVM heap in this setup. > > > > > I'm assuming that the system is using the rest of the RAM for disk > > > > cache. > > > > > Thanks for the great work! > > > > > Paul ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/search-performance/3255/6 "2017-07-06T04:20:20Z")