# Benchmark scripts/code? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 20, 2012, 12:26pm UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075 "2012-09-20T12:26:55Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![jdd](https://avatars.discourse-cdn.com/v4/letter/j/db5fbb/32.png) [@jdd](https://discuss.elastic.co/u/jdd)\ **Post date:** [September 20, 2012, 12:26pm UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/1 "2012-09-20T12:26:55Z")
Hi all, We have setup ES on a medium size cluster, and so far it's working smoothly (Thanks for that!) We wanted benchmark indexing and query performance, so wanted to check if there are any benchmarking tools out there for ES? (Googling did not help) Thanks, Jaideep -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 20, 2012, 12:38pm UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/2 "2012-09-20T12:38:23Z")
Hi Jaideep, I couldn't find anything either, thus I'm planning on writing a benchmark script for Elasticsearch in java for my own, but released in public on GitHub. I'll post my results in this group when I've got progress. Meanwhile you could stay in touch with my personal blog for progress. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/20 jaideep dhok [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > Hi all, > We have setup ES on a medium size cluster, and so far it's working > smoothly (Thanks for that!) > We wanted benchmark indexing and query performance, so wanted to check if > there are any benchmarking tools out there for ES? (Googling did not help) > > Thanks, > Jaideep > > -- -- ---
**Author:** ![jdd](https://avatars.discourse-cdn.com/v4/letter/j/db5fbb/32.png) [@jdd](https://discuss.elastic.co/u/jdd)\ **Post date:** [September 20, 2012, 1:23pm UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/3 "2012-09-20T13:23:51Z")
Robin, Do you have your code put up somewhere? Perhaps we can collaborate. Thanks, Jaideep On Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote: > Hi Jaideep, > > I couldn't find anything either, thus I'm planning on writing a benchmark > script for Elasticsearch in java for my own, but released in public on > GitHub. I'll post my results in this group when I've got progress. > Meanwhile you could stay in touch with my personal blog for progress. > > Best regards, > > Robin Verlangen > _Software engineer_ > \* > \* > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > E robin@us2.nl > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > 2012/9/20 jaideep dhok [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > > Hi all, > > We have setup ES on a medium size cluster, and so far it's working > > smoothly (Thanks for that!) > > We wanted benchmark indexing and query performance, so wanted to check if > > there are any benchmarking tools out there for ES? (Googling did not help) > > > > Thanks, > > Jaideep > > > > -- > > -- -- Jaideep Dhok -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 20, 2012, 1:28pm UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/4 "2012-09-20T13:28:48Z")
I'll try to post in on GitHub public ASAP. Probably within a couple of days. Let's stay in touch! Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/20 [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > Robin, > Do you have your code put up somewhere? Perhaps we can collaborate. > > Thanks, > Jaideep > > On Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote: > > > Hi Jaideep, > > > > I couldn't find anything either, thus I'm planning on writing a benchmark > > script for Elasticsearch in java for my own, but released in public on > > GitHub. I'll post my results in this group when I've got progress. > > Meanwhile you could stay in touch with my personal blog for progress. > > > > Best regards, > > > > Robin Verlangen > > _Software engineer_ > > \* > > \* > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > E robin@us2.nl > > > > Disclaimer: The information contained in this message and attachments is > > intended solely for the attention and use of the named addressee and may be > > confidential. If you are not the intended recipient, you are reminded that > > the information remains the property of the sender. You must not use, > > disclose, distribute, copy, print or rely on this e-mail. If you have > > received this message in error, please contact the sender immediately and > > irrevocably delete this message and any copies. > > > > 2012/9/20 jaideep dhok [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > > > > Hi all, > > > We have setup ES on a medium size cluster, and so far it's working > > > smoothly (Thanks for that!) > > > We wanted benchmark indexing and query performance, so wanted to check > > > if there are any benchmarking tools out there for ES? (Googling did not > > > help) > > > > > > Thanks, > > > Jaideep > > > > > > -- > > > > -- > > -- > Jaideep Dhok > > -- -- ---
**Author:** ![chenryn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chenryn/32/44917_2.png) [@chenryn](https://discuss.elastic.co/u/chenryn)\ **Post date:** [September 21, 2012, 6:59am UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/5 "2012-09-21T06:59:42Z")
I saw [http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html) but no comment space. How about the testing result? I try to aggregate my nginx access log into ES.but the index speed is bottleneck. 2012/9/20 Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) > I'll try to post in on GitHub public ASAP. Probably within a couple of > days. Let's stay in touch! > > Best regards, > > Robin Verlangen > _Software engineer_ > \* > \* > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > E robin@us2.nl > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > 2012/9/20 [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > Robin, > > > Do you have your code put up somewhere? Perhaps we can collaborate. > > > > Thanks, > > Jaideep > > > > On Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote: > > > > > Hi Jaideep, > > > > > > I couldn't find anything either, thus I'm planning on writing a > > > benchmark script for Elasticsearch in java for my own, but released in > > > public on GitHub. I'll post my results in this group when I've got > > > progress. Meanwhile you could stay in touch with my personal blog for > > > progress. > > > > > > Best regards, > > > > > > Robin Verlangen > > > _Software engineer_ > > > \* > > > \* > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > > E robin@us2.nl > > > > > > Disclaimer: The information contained in this message and attachments is > > > intended solely for the attention and use of the named addressee and may be > > > confidential. If you are not the intended recipient, you are reminded that > > > the information remains the property of the sender. You must not use, > > > disclose, distribute, copy, print or rely on this e-mail. If you have > > > received this message in error, please contact the sender immediately and > > > irrevocably delete this message and any copies. > > > > > > 2012/9/20 jaideep dhok [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > > > > > > Hi all, > > > > We have setup ES on a medium size cluster, and so far it's working > > > > smoothly (Thanks for that!) > > > > We wanted benchmark indexing and query performance, so wanted to check > > > > if there are any benchmarking tools out there for ES? (Googling did not > > > > help) > > > > > > > > Thanks, > > > > Jaideep > > > > > > > > -- > > > > > > -- > > > > -- > > Jaideep Dhok > > > > -- > > -- -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 21, 2012, 7:02am UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/6 "2012-09-21T07:02:43Z")
Hi Chenlin, Did you see the PDF file linked at the bottom of the post? We managed to get a throughput of 1250MB in under 10 minutes on a single node, with 10 parallel inserting threads. I'm not sure how big your logs are, but I think you should manage to do this. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/21 chenlin rao [rao.chenlin@gmail.com](mailto:rao.chenlin@gmail.com) > I saw > [http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html) but > no comment space. > How about the testing result? > I try to aggregate my nginx access log into ES.but the index speed > is bottleneck. > > 2012/9/20 Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) > > > I'll try to post in on GitHub public ASAP. Probably within a couple of > > days. Let's stay in touch! > > > > Best regards, > > > > Robin Verlangen > > _Software engineer_ > > \* > > \* > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > E robin@us2.nl > > > > Disclaimer: The information contained in this message and attachments is > > intended solely for the attention and use of the named addressee and may be > > confidential. If you are not the intended recipient, you are reminded that > > the information remains the property of the sender. You must not use, > > disclose, distribute, copy, print or rely on this e-mail. If you have > > received this message in error, please contact the sender immediately and > > irrevocably delete this message and any copies. > > > > 2012/9/20 [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > > > Robin, > > > > > Do you have your code put up somewhere? Perhaps we can collaborate. > > > > > > Thanks, > > > Jaideep > > > > > > On Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote: > > > > > > > Hi Jaideep, > > > > > > > > I couldn't find anything either, thus I'm planning on writing a > > > > benchmark script for Elasticsearch in java for my own, but released in > > > > public on GitHub. I'll post my results in this group when I've got > > > > progress. Meanwhile you could stay in touch with my personal blog for > > > > progress. > > > > > > > > Best regards, > > > > > > > > Robin Verlangen > > > > _Software engineer_ > > > > \* > > > > \* > > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > > > E robin@us2.nl > > > > > > > > Disclaimer: The information contained in this message and attachments > > > > is intended solely for the attention and use of the named addressee and may > > > > be confidential. If you are not the intended recipient, you are reminded > > > > that the information remains the property of the sender. You must not use, > > > > disclose, distribute, copy, print or rely on this e-mail. If you have > > > > received this message in error, please contact the sender immediately and > > > > irrevocably delete this message and any copies. > > > > > > > > 2012/9/20 jaideep dhok [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > > > > > > > > Hi all, > > > > > We have setup ES on a medium size cluster, and so far it's working > > > > > smoothly (Thanks for that!) > > > > > We wanted benchmark indexing and query performance, so wanted to check > > > > > if there are any benchmarking tools out there for ES? (Googling did not > > > > > help) > > > > > > > > > > Thanks, > > > > > Jaideep > > > > > > > > > > -- > > > > > > > > -- > > > > > > -- > > > Jaideep Dhok > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![chenryn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chenryn/32/44917_2.png) [@chenryn](https://discuss.elastic.co/u/chenryn)\ **Post date:** [September 21, 2012, 9:53am UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/7 "2012-09-21T09:53:41Z")
tks~ In my testing(a perl script bulk index nginx access log using http protocol ), the count / time ~ 5000 rps when I fork four porcesses. But the CPU Util increase to 300%+. And after I add the second node using default config, the index rps decrease to 2500! Now I am feeling uncertain about this... 2012/9/21 Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) > Hi Chenlin, > > Did you see the PDF file linked at the bottom of the post? We managed to > get a throughput of 1250MB in under 10 minutes on a single node, with 10 > parallel inserting threads. I'm not sure how big your logs are, but I think > you should manage to do this. > > Best regards, > > Robin Verlangen > _Software engineer_ > \* > \* > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > E robin@us2.nl > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > 2012/9/21 chenlin rao [rao.chenlin@gmail.com](mailto:rao.chenlin@gmail.com) > > > I saw > > [http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html) but > > no comment space. > > How about the testing result? > > I try to aggregate my nginx access log into ES.but the index speed > > is bottleneck. > > > > 2012/9/20 Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) > > > > > I'll try to post in on GitHub public ASAP. Probably within a couple of > > > days. Let's stay in touch! > > > > > > Best regards, > > > > > > Robin Verlangen > > > _Software engineer_ > > > \* > > > \* > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > > E robin@us2.nl > > > > > > Disclaimer: The information contained in this message and attachments is > > > intended solely for the attention and use of the named addressee and may be > > > confidential. If you are not the intended recipient, you are reminded that > > > the information remains the property of the sender. You must not use, > > > disclose, distribute, copy, print or rely on this e-mail. If you have > > > received this message in error, please contact the sender immediately and > > > irrevocably delete this message and any copies. > > > > > > 2012/9/20 [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > > > > > Robin, > > > > > > > Do you have your code put up somewhere? Perhaps we can collaborate. > > > > > > > > Thanks, > > > > Jaideep > > > > > > > > On Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote: > > > > > > > > > Hi Jaideep, > > > > > > > > > > I couldn't find anything either, thus I'm planning on writing a > > > > > benchmark script for Elasticsearch in java for my own, but released in > > > > > public on GitHub. I'll post my results in this group when I've got > > > > > progress. Meanwhile you could stay in touch with my personal blog for > > > > > progress. > > > > > > > > > > Best regards, > > > > > > > > > > Robin Verlangen > > > > > _Software engineer_ > > > > > \* > > > > > \* > > > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > > > > E robin@us2.nl > > > > > > > > > > Disclaimer: The information contained in this message and attachments > > > > > is intended solely for the attention and use of the named addressee and may > > > > > be confidential. If you are not the intended recipient, you are reminded > > > > > that the information remains the property of the sender. You must not use, > > > > > disclose, distribute, copy, print or rely on this e-mail. If you have > > > > > received this message in error, please contact the sender immediately and > > > > > irrevocably delete this message and any copies. > > > > > > > > > > 2012/9/20 jaideep dhok [jaideep.dhok@gmail.com](mailto:jaideep.dhok@gmail.com) > > > > > > > > > > > Hi all, > > > > > > We have setup ES on a medium size cluster, and so far it's working > > > > > > smoothly (Thanks for that!) > > > > > > We wanted benchmark indexing and query performance, so wanted to > > > > > > check if there are any benchmarking tools out there for ES? (Googling did > > > > > > not help) > > > > > > > > > > > > Thanks, > > > > > > Jaideep > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > -- > > > > Jaideep Dhok > > > > > > > > -- > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [September 21, 2012, 10:00am UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/8 "2012-09-21T10:00:40Z")
Hi Chenlin > In my testing(a perl script bulk index nginx access log using http > protocol ), the count / time ~ 5000 rps when I fork four porcesses. > But the CPU Util increase to 300%+. > And after I add the second node using default config, the index rps > decrease to 2500! Now I am feeling uncertain about this... You don't provide any information about whether you are running these two nodes on different servers, or on the same machine where you're running the indexing script, or if the servers have different configurations etc etc. clint > -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 21, 2012, 10:24am UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/9 "2012-09-21T10:24:38Z")
Nothing in-depth about this, but we managed to get roughly 20k indexes / sec with 12 simultaneous threads (java code) on 1 node, running on a 16GB RAM, quad core, 1x 240GB SSD machine. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/21 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > Hi Chenlin > > > In my testing(a perl script bulk index nginx access log using http > > protocol ), the count / time ~ 5000 rps when I fork four porcesses. > > But the CPU Util increase to 300%+. > > And after I add the second node using default config, the index rps > > decrease to 2500! Now I am feeling uncertain about this... > > You don't provide any information about whether you are running these > two nodes on different servers, or on the same machine where you're > running the indexing script, or if the servers have different > configurations etc etc. > > clint > > > > > -- -- ---
**Author:** ![chenryn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chenryn/32/44917_2.png) [@chenryn](https://discuss.elastic.co/u/chenryn)\ **Post date:** [September 21, 2012, 10:38am UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/10 "2012-09-21T10:38:10Z")
sorry~ These two nodes run on two servers whose configurations are same(4 \* E5606/48G mem/3T sata/default ES.config). But my indexing script run on one of them. 2012/9/21 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > Hi Chenlin > > > In my testing(a perl script bulk index nginx access log using http > > protocol ), the count / time ~ 5000 rps when I fork four porcesses. > > But the CPU Util increase to 300%+. > > And after I add the second node using default config, the index rps > > decrease to 2500! Now I am feeling uncertain about this... > > You don't provide any information about whether you are running these > two nodes on different servers, or on the same machine where you're > running the indexing script, or if the servers have different > configurations etc etc. > > clint > > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:12am UTC](https://discuss.elastic.co/t/benchmark-scripts-code/9075/11 "2017-07-06T03:12:00Z")