# \# of shards and # of data paths(striping) **URL:** \ **Category:** Elasticsearch\ **Created:** [April 19, 2012, 11:52am UTC](https://discuss.elastic.co/t/of-shards-and-of-data-paths-striping/7396 "2012-04-19T11:52:36Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![md\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/md_2/32/2896_2.png) [@md\_2](https://discuss.elastic.co/u/md_2)\ **Post date:** [April 19, 2012, 11:52am UTC](https://discuss.elastic.co/t/of-shards-and-of-data-paths-striping/7396/1 "2012-04-19T11:52:36Z")
Hi, Has anyone tried indexing performance measurement between having more shards and more data paths? My question comes from my wondering if having more data paths might become cost effective on EC2. For example, consider two configurations(5 shards, 0 replica for simplicity). A: Standard Large instance with 5 EBS(8GB) volumes B: 5 \* Standard Small instance with 1 EBS(8GB) volume Standard Large is exactly 4 times better and expensive as Standard Small. But A costs less because one more EBS(4 more to be exact) costs less then one more Standard Small instance. In traditional RDBMS, fsync-heavy database operations benefit a LOT from more volumes. So it might have been clear that A outperforms B. But ES does as few fsyncs(commits) as possible according to the video, "Road to a Distributed Search Engine". So, how do they perform? I appreciate your thoughts and ideas. Also, does it make sense to have shared benchmark for ES? Or already have one? Thanks, Takenori ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [April 19, 2012, 1:00pm UTC](https://discuss.elastic.co/t/of-shards-and-of-data-paths-striping/7396/2 "2012-04-19T13:00:53Z")
Another thing to consider when running ES on AWS is the IO. AWS small instances have low I/O performance, hence option B would likely not perform well. Small instances are also more susceptible to noisy neighbor issues. Just my 3.14 cents... Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Thu, Apr 19, 2012 at 7:52 AM, Sato Takenori [takenori.sato@gmail.com](mailto:takenori.sato@gmail.com)wrote: > Hi, > > Has anyone tried indexing performance measurement between having more > shards and more data paths? > > My question comes from my wondering if having more data paths might become > cost effective on EC2. > > For example, consider two configurations(5 shards, 0 replica for > simplicity). > > A: Standard Large instance with 5 EBS(8GB) volumes > B: 5 \* Standard Small instance with 1 EBS(8GB) volume > > Standard Large is exactly 4 times better and expensive as Standard Small. > But A costs less because one more EBS(4 more to be exact) costs less then > one more Standard Small instance. > > In traditional RDBMS, fsync-heavy database operations benefit a LOT from > more volumes. > So it might have been clear that A outperforms B. > > But ES does as few fsyncs(commits) as possible according to the video, > "Road to a Distributed Search Engine". > So, how do they perform? > > I appreciate your thoughts and ideas. > > Also, does it make sense to have shared benchmark for ES? Or already have > one? > > Thanks, > Takenori ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [April 19, 2012, 7:40pm UTC](https://discuss.elastic.co/t/of-shards-and-of-data-paths-striping/7396/3 "2012-04-19T19:40:19Z")
Hi Takenori, Yes, I think this is more or less a known things - that more EBS volumes are better than one because you never know when one of them will "have a bad day". Plus small EC2 instances can have poor IO. But all this is hard/impossible to measure precisely because you never know what exactly you are measuring, who/what else is interfering with your benchmarking, etc. ## Otis Performance Monitoring SaaS for Elasticsearch - > **[Sematext Monitoring | Infrastructure Monitoring Service](https://sematext.com/spm/)** > > Bring together your servers, Apps, Metrics, Logs & Events ✓ Over 40 integrations to easily collect metrics & events across your whole stack ✓ Start now! On Thursday, April 19, 2012 7:52:36 AM UTC-4, Sato Takenori wrote: > Hi, > > Has anyone tried indexing performance measurement between having more > shards and more data paths? > > My question comes from my wondering if having more data paths might become > cost effective on EC2. > > For example, consider two configurations(5 shards, 0 replica for > simplicity). > > A: Standard Large instance with 5 EBS(8GB) volumes > B: 5 \* Standard Small instance with 1 EBS(8GB) volume > > Standard Large is exactly 4 times better and expensive as Standard Small. > But A costs less because one more EBS(4 more to be exact) costs less then > one more Standard Small instance. > > In traditional RDBMS, fsync-heavy database operations benefit a LOT from > more volumes. > So it might have been clear that A outperforms B. > > But ES does as few fsyncs(commits) as possible according to the video, > "Road to a Distributed Search Engine". > So, how do they perform? > > I appreciate your thoughts and ideas. > > Also, does it make sense to have shared benchmark for ES? Or already have > one? > > Thanks, > Takenori ---
**Author:** ![md\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/md_2/32/2896_2.png) [@md\_2](https://discuss.elastic.co/u/md_2)\ **Post date:** [April 20, 2012, 1:13am UTC](https://discuss.elastic.co/t/of-shards-and-of-data-paths-striping/7396/4 "2012-04-20T01:13:38Z")
Thanks Berkay and Otis! It may not be obvious, but Small instance sounds unstable as bad as micro. I will start evaluating Large instance cluster(with EBS volumes) instead of Small ones. In the "Deploying Elasticsearch with Chef Solo", Large instance is also recommended for testing. Best, Takenori 2012年4月20日金曜日 4時40分19秒 UTC+9 Otis Gospodnetic: > Hi Takenori, > > Yes, I think this is more or less a known things - that more EBS volumes > are better than one because you never know when one of them will "have a > bad day". > Plus small EC2 instances can have poor IO. > > But all this is hard/impossible to measure precisely because you never > know what exactly you are measuring, who/what else is interfering with your > benchmarking, etc. > > ## Otis > > Performance Monitoring SaaS for Elasticsearch - > [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Thursday, April 19, 2012 7:52:36 AM UTC-4, Sato Takenori wrote: > > > Hi, > > > > Has anyone tried indexing performance measurement between having more > > shards and more data paths? > > > > My question comes from my wondering if having more data paths might > > become cost effective on EC2. > > > > For example, consider two configurations(5 shards, 0 replica for > > simplicity). > > > > A: Standard Large instance with 5 EBS(8GB) volumes > > B: 5 \* Standard Small instance with 1 EBS(8GB) volume > > > > Standard Large is exactly 4 times better and expensive as Standard Small. > > But A costs less because one more EBS(4 more to be exact) costs less then > > one more Standard Small instance. > > > > In traditional RDBMS, fsync-heavy database operations benefit a LOT from > > more volumes. > > So it might have been clear that A outperforms B. > > > > But ES does as few fsyncs(commits) as possible according to the video, > > "Road to a Distributed Search Engine". > > So, how do they perform? > > > > I appreciate your thoughts and ideas. > > > > Also, does it make sense to have shared benchmark for ES? Or already have > > one? > > > > Thanks, > > Takenori ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:31am UTC](https://discuss.elastic.co/t/of-shards-and-of-data-paths-striping/7396/5 "2017-07-06T03:31:46Z")
Closed on July 6, 2017, 3:31am UTC