Hi all,
We have setup ES on a medium size cluster, and so far it's working smoothly
(Thanks for that!)
We wanted benchmark indexing and query performance, so wanted to check if
there are any benchmarking tools out there for ES? (Googling did not help)
Thanks,
Jaideep
--
Hi Jaideep,
I couldn't find anything either, thus I'm planning on writing a benchmark
script for Elasticsearch in java for my own, but released in public on
GitHub. I'll post my results in this group when I've got progress.
Meanwhile you could stay in touch with my personal blog for progress.
Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/20 jaideep dhok jaideep.dhok@gmail.com
Hi all,
We have setup ES on a medium size cluster, and so far it's working
smoothly (Thanks for that!)
We wanted benchmark indexing and query performance, so wanted to check if
there are any benchmarking tools out there for ES? (Googling did not help)Thanks,
Jaideep--
--
Robin,
Do you have your code put up somewhere? Perhaps we can collaborate.
Thanks,
Jaideep
On Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen robin@us2.nl wrote:
Hi Jaideep,
I couldn't find anything either, thus I'm planning on writing a benchmark
script for Elasticsearch in java for my own, but released in public on
GitHub. I'll post my results in this group when I've got progress.
Meanwhile you could stay in touch with my personal blog for progress.Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep dhok jaideep.dhok@gmail.com
Hi all,
We have setup ES on a medium size cluster, and so far it's working
smoothly (Thanks for that!)
We wanted benchmark indexing and query performance, so wanted to check if
there are any benchmarking tools out there for ES? (Googling did not help)Thanks,
Jaideep--
--
--
Jaideep Dhok
--
I'll try to post in on GitHub public ASAP. Probably within a couple of
days. Let's stay in touch!
Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/20 jaideep.dhok@gmail.com jaideep.dhok@gmail.com
Robin,
Do you have your code put up somewhere? Perhaps we can collaborate.Thanks,
JaideepOn Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen robin@us2.nl wrote:
Hi Jaideep,
I couldn't find anything either, thus I'm planning on writing a benchmark
script for Elasticsearch in java for my own, but released in public on
GitHub. I'll post my results in this group when I've got progress.
Meanwhile you could stay in touch with my personal blog for progress.Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep dhok jaideep.dhok@gmail.com
Hi all,
We have setup ES on a medium size cluster, and so far it's working
smoothly (Thanks for that!)
We wanted benchmark indexing and query performance, so wanted to check
if there are any benchmarking tools out there for ES? (Googling did not
help)Thanks,
Jaideep--
--
--
Jaideep Dhok--
--
I saw
http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html
but
no comment space.
How about the testing result?
I try to aggregate my nginx access log into ES.but the index speed
is bottleneck.
2012/9/20 Robin Verlangen robin@us2.nl
I'll try to post in on GitHub public ASAP. Probably within a couple of
days. Let's stay in touch!Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep.dhok@gmail.com jaideep.dhok@gmail.com
Robin,
Do you have your code put up somewhere? Perhaps we can collaborate.
Thanks,
JaideepOn Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen robin@us2.nl wrote:
Hi Jaideep,
I couldn't find anything either, thus I'm planning on writing a
benchmark script for Elasticsearch in java for my own, but released in
public on GitHub. I'll post my results in this group when I've got
progress. Meanwhile you could stay in touch with my personal blog for
progress.Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep dhok jaideep.dhok@gmail.com
Hi all,
We have setup ES on a medium size cluster, and so far it's working
smoothly (Thanks for that!)
We wanted benchmark indexing and query performance, so wanted to check
if there are any benchmarking tools out there for ES? (Googling did not
help)Thanks,
Jaideep--
--
--
Jaideep Dhok--
--
--
Hi Chenlin,
Did you see the PDF file linked at the bottom of the post? We managed to
get a throughput of 1250MB in under 10 minutes on a single node, with 10
parallel inserting threads. I'm not sure how big your logs are, but I think
you should manage to do this.
Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/21 chenlin rao rao.chenlin@gmail.com
I saw
http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html but
no comment space.
How about the testing result?
I try to aggregate my nginx access log into ES.but the index speed
is bottleneck.2012/9/20 Robin Verlangen robin@us2.nl
I'll try to post in on GitHub public ASAP. Probably within a couple of
days. Let's stay in touch!Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep.dhok@gmail.com jaideep.dhok@gmail.com
Robin,
Do you have your code put up somewhere? Perhaps we can collaborate.
Thanks,
JaideepOn Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen robin@us2.nl wrote:
Hi Jaideep,
I couldn't find anything either, thus I'm planning on writing a
benchmark script for Elasticsearch in java for my own, but released in
public on GitHub. I'll post my results in this group when I've got
progress. Meanwhile you could stay in touch with my personal blog for
progress.Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments
is intended solely for the attention and use of the named addressee and may
be confidential. If you are not the intended recipient, you are reminded
that the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep dhok jaideep.dhok@gmail.com
Hi all,
We have setup ES on a medium size cluster, and so far it's working
smoothly (Thanks for that!)
We wanted benchmark indexing and query performance, so wanted to check
if there are any benchmarking tools out there for ES? (Googling did not
help)Thanks,
Jaideep--
--
--
Jaideep Dhok--
--
--
--
tks~
In my testing(a perl script bulk index nginx access log using http protocol
), the count / time ~ 5000 rps when I fork four porcesses. But the CPU
Util increase to 300%+.
And after I add the second node using default config, the index rps
decrease to 2500! Now I am feeling uncertain about this...
2012/9/21 Robin Verlangen robin@us2.nl
Hi Chenlin,
Did you see the PDF file linked at the bottom of the post? We managed to
get a throughput of 1250MB in under 10 minutes on a single node, with 10
parallel inserting threads. I'm not sure how big your logs are, but I think
you should manage to do this.Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/21 chenlin rao rao.chenlin@gmail.com
I saw
http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html but
no comment space.
How about the testing result?
I try to aggregate my nginx access log into ES.but the index speed
is bottleneck.2012/9/20 Robin Verlangen robin@us2.nl
I'll try to post in on GitHub public ASAP. Probably within a couple of
days. Let's stay in touch!Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep.dhok@gmail.com jaideep.dhok@gmail.com
Robin,
Do you have your code put up somewhere? Perhaps we can collaborate.
Thanks,
JaideepOn Thu, Sep 20, 2012 at 6:08 PM, Robin Verlangen robin@us2.nl wrote:
Hi Jaideep,
I couldn't find anything either, thus I'm planning on writing a
benchmark script for Elasticsearch in java for my own, but released in
public on GitHub. I'll post my results in this group when I've got
progress. Meanwhile you could stay in touch with my personal blog for
progress.Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nlDisclaimer: The information contained in this message and attachments
is intended solely for the attention and use of the named addressee and may
be confidential. If you are not the intended recipient, you are reminded
that the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.2012/9/20 jaideep dhok jaideep.dhok@gmail.com
Hi all,
We have setup ES on a medium size cluster, and so far it's working
smoothly (Thanks for that!)
We wanted benchmark indexing and query performance, so wanted to
check if there are any benchmarking tools out there for ES? (Googling did
not help)Thanks,
Jaideep--
--
--
Jaideep Dhok--
--
--
--
--
Hi Chenlin
In my testing(a perl script bulk index nginx access log using http
protocol ), the count / time ~ 5000 rps when I fork four porcesses.
But the CPU Util increase to 300%+.
And after I add the second node using default config, the index rps
decrease to 2500! Now I am feeling uncertain about this...
You don't provide any information about whether you are running these
two nodes on different servers, or on the same machine where you're
running the indexing script, or if the servers have different
configurations etc etc.
clint
--
Nothing in-depth about this, but we managed to get roughly 20k indexes /
sec with 12 simultaneous threads (java code) on 1 node, running on a 16GB
RAM, quad core, 1x 240GB SSD machine.
Best regards,
Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/21 Clinton Gormley clint@traveljury.com
Hi Chenlin
In my testing(a perl script bulk index nginx access log using http
protocol ), the count / time ~ 5000 rps when I fork four porcesses.
But the CPU Util increase to 300%+.
And after I add the second node using default config, the index rps
decrease to 2500! Now I am feeling uncertain about this...You don't provide any information about whether you are running these
two nodes on different servers, or on the same machine where you're
running the indexing script, or if the servers have different
configurations etc etc.clint
--
--
sorry~
These two nodes run on two servers whose configurations are same(4 *
E5606/48G mem/3T sata/default ES.config). But my indexing script run on one
of them.
2012/9/21 Clinton Gormley clint@traveljury.com
Hi Chenlin
In my testing(a perl script bulk index nginx access log using http
protocol ), the count / time ~ 5000 rps when I fork four porcesses.
But the CPU Util increase to 300%+.
And after I add the second node using default config, the index rps
decrease to 2500! Now I am feeling uncertain about this...You don't provide any information about whether you are running these
two nodes on different servers, or on the same machine where you're
running the indexing script, or if the servers have different
configurations etc etc.clint
--
--
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.