# Snapshot Scaling Problems **URL:** \ **Category:** Elasticsearch\ **Created:** [March 9, 2015, 6:18pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586 "2015-03-09T18:18:11Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Andy\_Nemzek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_nemzek/32/776_2.png) [@Andy\_Nemzek](https://discuss.elastic.co/u/Andy_Nemzek)\ **Post date:** [March 9, 2015, 6:18pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/1 "2015-03-09T18:18:11Z")
Hello, My company is using the ELK stack. Right now we have a very small amount of data actually being sent to elastic search (probably a couple hundred logstash entries a day if that), however, the data that is getting logged is very important. I recently set up snapshots to help protect this data. I take 1 snapshot a day, I delete snapshots that are older than 20 days, and each snapshot is comprised of all the logstash indexes in elasticsearch. It's also a business requirement that we are able to search at least a year's worth of data, so I can't close logstash indexes unless they're older than at least a year. Now, we've been using logstash for several months and each day it creates a new index. We've found that even though there is very little data in these indexes, it's taking upwards of 30 minutes to take a snapshot of all of them and each day it appears to take 20 - 100 seconds longer than the last. It is also taking about 30 minutes to delete a single snapshot, which is done each day as a part of cleaning up old snapshots. So, the whole process is is taking about an hour each day and appears to be growing longer very quickly. Am I doing something wrong here or is this kind of thing expected? It's seems pretty strange that it's taking so long with the little amount of data we have. I've looked through the snapshot docs several times and there doesn't appear to be much talk about how the process scales. Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 9, 2015, 6:58pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/2 "2015-03-09T18:58:06Z")
How many indices are there, are you using the default shard count (5)? Are you optimising older indices? The snapshot takes segments, so it may be that there is a lot of them to copy. You could try optimising your old indices, eg older than 7 days, down to a single segment and then see if that helps. Be aware though, the optimise is a resource heavy operation, so unless you have a lot of resources you should only run one at a time. On 10 March 2015 at 05:18, Andy Nemzek [bitknower@gmail.com](mailto:bitknower@gmail.com) wrote: > Hello, > > My company is using the ELK stack. Right now we have a very small amount > of data actually being sent to Elasticsearch (probably a couple hundred > logstash entries a day if that), however, the data that is getting logged > is very important. I recently set up snapshots to help protect this data. > > I take 1 snapshot a day, I delete snapshots that are older than 20 days, > and each snapshot is comprised of all the logstash indexes in > elasticsearch. It's also a business requirement that we are able to search > at least a year's worth of data, so I can't close logstash indexes unless > they're older than at least a year. > > Now, we've been using logstash for several months and each day it creates > a new index. We've found that even though there is very little data in > these indexes, it's taking upwards of 30 minutes to take a snapshot of all > of them and each day it appears to take 20 - 100 seconds longer than the > last. It is also taking about 30 minutes to delete a single snapshot, > which is done each day as a part of cleaning up old snapshots. So, the > whole process is is taking about an hour each day and appears to be growing > longer very quickly. > > Am I doing something wrong here or is this kind of thing expected? It's > seems pretty strange that it's taking so long with the little amount of > data we have. I've looked through the snapshot docs several times and > there doesn't appear to be much talk about how the process scales. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-wCPeyx7mYkan7sN0BK8suHY3RR7bCU19dN0Qn%3DpyALA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-wCPeyx7mYkan7sN0BK8suHY3RR7bCU19dN0Qn%3DpyALA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andy\_Nemzek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_nemzek/32/776_2.png) [@Andy\_Nemzek](https://discuss.elastic.co/u/Andy_Nemzek)\ **Post date:** [March 9, 2015, 7:29pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/3 "2015-03-09T19:29:58Z")
Hi Mark, Thanks for the reply. We've been using logstash for several months now and it creates a new index each day, so I imagine there are over 100 indexes at this point. Elasticsearch is running on a single machine...I haven't done anything with shards, so the defaults must be in use. Haven't optimized old indexes. We're pretty much just running ELK out of the box. When you mention 'optimizing indexes', does this process combine indexes? Do you know if these performance problems are typical when using ELK out of the box? On Monday, March 9, 2015 at 1:58:33 PM UTC-5, Mark Walkom wrote: > How many indices are there, are you using the default shard count (5)? Are > you optimising older indices? > > The snapshot takes segments, so it may be that there is a lot of them to > copy. You could try optimising your old indices, eg older than 7 days, down > to a single segment and then see if that helps. > > Be aware though, the optimise is a resource heavy operation, so unless you > have a lot of resources you should only run one at a time. > > On 10 March 2015 at 05:18, Andy Nemzek \<[bitk...@gmail.com](mailto:bitk...@gmail.com) \\> > wrote: > > > Hello, > > > > My company is using the ELK stack. Right now we have a very small amount > > of data actually being sent to Elasticsearch (probably a couple hundred > > logstash entries a day if that), however, the data that is getting logged > > is very important. I recently set up snapshots to help protect this data. > > > > I take 1 snapshot a day, I delete snapshots that are older than 20 days, > > and each snapshot is comprised of all the logstash indexes in > > elasticsearch. It's also a business requirement that we are able to search > > at least a year's worth of data, so I can't close logstash indexes unless > > they're older than at least a year. > > > > Now, we've been using logstash for several months and each day it creates > > a new index. We've found that even though there is very little data in > > these indexes, it's taking upwards of 30 minutes to take a snapshot of all > > of them and each day it appears to take 20 - 100 seconds longer than the > > last. It is also taking about 30 minutes to delete a single snapshot, > > which is done each day as a part of cleaning up old snapshots. So, the > > whole process is is taking about an hour each day and appears to be growing > > longer very quickly. > > > > Am I doing something wrong here or is this kind of thing expected? It's > > seems pretty strange that it's taking so long with the little amount of > > data we have. I've looked through the snapshot docs several times and > > there doesn't appear to be much talk about how the process scales. > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/13241ad4-9ae4-4ac6-b5e9-421a5c62b898%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13241ad4-9ae4-4ac6-b5e9-421a5c62b898%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andy\_Nemzek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_nemzek/32/776_2.png) [@Andy\_Nemzek](https://discuss.elastic.co/u/Andy_Nemzek)\ **Post date:** [March 9, 2015, 7:59pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/4 "2015-03-09T19:59:48Z")
I forgot to mention that we're also using the S3 snapshot plugin to store the snapshots in an S3 bucket. Perhaps this might be part of the performance problems? On Monday, March 9, 2015 at 1:58:33 PM UTC-5, Mark Walkom wrote: > How many indices are there, are you using the default shard count (5)? Are > you optimising older indices? > > The snapshot takes segments, so it may be that there is a lot of them to > copy. You could try optimising your old indices, eg older than 7 days, down > to a single segment and then see if that helps. > > Be aware though, the optimise is a resource heavy operation, so unless you > have a lot of resources you should only run one at a time. > > On 10 March 2015 at 05:18, Andy Nemzek \<[bitk...@gmail.com](mailto:bitk...@gmail.com) \\> > wrote: > > > Hello, > > > > My company is using the ELK stack. Right now we have a very small amount > > of data actually being sent to Elasticsearch (probably a couple hundred > > logstash entries a day if that), however, the data that is getting logged > > is very important. I recently set up snapshots to help protect this data. > > > > I take 1 snapshot a day, I delete snapshots that are older than 20 days, > > and each snapshot is comprised of all the logstash indexes in > > elasticsearch. It's also a business requirement that we are able to search > > at least a year's worth of data, so I can't close logstash indexes unless > > they're older than at least a year. > > > > Now, we've been using logstash for several months and each day it creates > > a new index. We've found that even though there is very little data in > > these indexes, it's taking upwards of 30 minutes to take a snapshot of all > > of them and each day it appears to take 20 - 100 seconds longer than the > > last. It is also taking about 30 minutes to delete a single snapshot, > > which is done each day as a part of cleaning up old snapshots. So, the > > whole process is is taking about an hour each day and appears to be growing > > longer very quickly. > > > > Am I doing something wrong here or is this kind of thing expected? It's > > seems pretty strange that it's taking so long with the little amount of > > data we have. I've looked through the snapshot docs several times and > > there doesn't appear to be much talk about how the process scales. > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0add3377-4b49-4a82-a233-e005113ab1b9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e5844cef-e8e2-4822-b77b-b6606b409eb8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e5844cef-e8e2-4822-b77b-b6606b409eb8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 11, 2015, 7:11am UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/5 "2015-03-11T07:11:50Z")
On Monday, March 09, 2015 at 20:29 CET, Andy Nemzek [bitknower@gmail.com](mailto:bitknower@gmail.com) wrote: > We've been using logstash for several months now and it creates a new > index each day, so I imagine there are over 100 indexes at this point. Why create daily indexes if you only have a few hundred entries in each? There's a constant overhead for each shard so you don't want more indexes than you need. Seems like you'd be fine with montly indexes, and then your snapshot problems would disappear too. > Elasticsearch is running on a single machine...I haven't done anything > with shards, so the defaults must be in use. Haven't optimized old > indexes. We're pretty much just running ELK out of the box. When you > mention 'optimizing indexes', does this process combine indexes? No, but it can combine segments in a Lucene index (that make up Elasticsearch indexes), and segments are what's being backed up. So the more segments you have the the longer time snapshots are going to take. > Do you know if these performance problems are typical when > using ELK out of the box? 100 indexes on a single box should be okay but it depends on the size of the JVM heap. -- Magnus Bäck | Software Engineer, Development Tools [magnus.back@sonymobile.com](mailto:magnus.back@sonymobile.com) | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20150311071150.GB5729%40seldlx20533.corpusers.net](https://groups.google.com/d/msgid/elasticsearch/20150311071150.GB5729%40seldlx20533.corpusers.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Aaron\_Mefford](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@Aaron\_Mefford](https://discuss.elastic.co/u/Aaron_Mefford)\ **Post date:** [March 12, 2015, 5:50pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/6 "2015-03-12T17:50:22Z")
With the low volume of ingest, and the long duration of history, Id suggest you may want to trim back the number of shards per index from the default 5. Based on your 100 docs per day Id say 1 shard per day. If you combined this with the other suggestion to increase the duration of an index, then you might increase the number of shards, but maybe still not. Running an optimize once you have completed a time period is great advice if you can afford the overhead, sounds like one day at a time you should be able to, and that the overhead of not optimizing is costing you more when you snapshot. And index is made of shards, a shard is made of lucene segments. Lucene segments are the actual files that you copy when you snapshot. As such the number of segments is multiplied by the number of shards per index and the number of indexes. Reducing the number of indexes by creating larger time periods will significantly reduce the number of segments. Reducing the number of shards per index will significantly reduce the number of segments. Optimizing the index will also consolidate many segments into a single segment. Based on the use of S3 should we assume you are using AWS EC2? What instance size? Your data volume seems very low so it seems concerning that you have such a large time period to snapshot, and points to a slow file system, or a significant number of segments (100 indexes, 5 shards per index, xx segments per shard, == many thousands of segments). What does your storage system look like? If you are using EC2 are you using the newer EBS volumes (SSD backed)? Some of the smaller instance size significantly limit prolonged EBS throughput, in my experience. On Wednesday, March 11, 2015 at 1:12:01 AM UTC-6, Magnus Bäck wrote: > On Monday, March 09, 2015 at 20:29 CET, > Andy Nemzek \<[bitk...@gmail.com](mailto:bitk...@gmail.com) \\> wrote: > > > We've been using logstash for several months now and it creates a new > > index each day, so I imagine there are over 100 indexes at this point. > > Why create daily indexes if you only have a few hundred entries in each? > There's a constant overhead for each shard so you don't want more > indexes than you need. Seems like you'd be fine with montly indexes, > and then your snapshot problems would disappear too. > > > Elasticsearch is running on a single machine...I haven't done anything > > with shards, so the defaults must be in use. Haven't optimized old > > indexes. We're pretty much just running ELK out of the box. When you > > mention 'optimizing indexes', does this process combine indexes? > > No, but it can combine segments in a Lucene index (that make up > Elasticsearch indexes), and segments are what's being backed up. > So the more segments you have the the longer time snapshots are > going to take. > > > Do you know if these performance problems are typical when > > using ELK out of the box? > > 100 indexes on a single box should be okay but it depends on > the size of the JVM heap. > > -- > Magnus Bäck | Software Engineer, Development Tools > [magnu...@sonymobile.com](mailto:magnu...@sonymobile.com) \ | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7be4c805-b4f1-424d-b67b-2ad70e5da659%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7be4c805-b4f1-424d-b67b-2ad70e5da659%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andy\_Nemzek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_nemzek/32/776_2.png) [@Andy\_Nemzek](https://discuss.elastic.co/u/Andy_Nemzek)\ **Post date:** [March 13, 2015, 4:30am UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/7 "2015-03-13T04:30:02Z")
Thank you guys for your thoughts here. This is really useful information. Again, we're creating daily indexes because that's what logstash does out of the box with the elasticsearch plugin, and this kind of tuning info isn't included with that plugin. Minimizing both the number of indexes and shards now sounds like a great idea. We are indeed using EC2. We're just using an m1.small that's EBS backed (non-SSD). So, yes, it's not a very powerful machine, but again, we're not throwing a lot of data at it either. On Thursday, March 12, 2015 at 12:50:22 PM UTC-5, [aa...@definemg.com](mailto:aa...@definemg.com) wrote: > With the low volume of ingest, and the long duration of history, Id > suggest you may want to trim back the number of shards per index from the > default 5. Based on your 100 docs per day Id say 1 shard per day. If you > combined this with the other suggestion to increase the duration of an > index, then you might increase the number of shards, but maybe still not. > Running an optimize once you have completed a time period is great advice > if you can afford the overhead, sounds like one day at a time you should be > able to, and that the overhead of not optimizing is costing you more when > you snapshot. > > And index is made of shards, a shard is made of lucene segments. Lucene > segments are the actual files that you copy when you snapshot. As such the > number of segments is multiplied by the number of shards per index and the > number of indexes. Reducing the number of indexes by creating larger time > periods will significantly reduce the number of segments. Reducing the > number of shards per index will significantly reduce the number of > segments. Optimizing the index will also consolidate many segments into a > single segment. > > Based on the use of S3 should we assume you are using AWS EC2? What > instance size? Your data volume seems very low so it seems concerning that > you have such a large time period to snapshot, and points to a slow file > system, or a significant number of segments (100 indexes, 5 shards per > index, xx segments per shard, == many thousands of segments). What does > your storage system look like? If you are using EC2 are you using the > newer EBS volumes (SSD backed)? Some of the smaller instance size > significantly limit prolonged EBS throughput, in my experience. > > On Wednesday, March 11, 2015 at 1:12:01 AM UTC-6, Magnus Bäck wrote: > > > On Monday, March 09, 2015 at 20:29 CET, > > Andy Nemzek [bitk...@gmail.com](mailto:bitk...@gmail.com) wrote: > > > > > We've been using logstash for several months now and it creates a new > > > index each day, so I imagine there are over 100 indexes at this point. > > > > Why create daily indexes if you only have a few hundred entries in each? > > There's a constant overhead for each shard so you don't want more > > indexes than you need. Seems like you'd be fine with montly indexes, > > and then your snapshot problems would disappear too. > > > > > Elasticsearch is running on a single machine...I haven't done anything > > > with shards, so the defaults must be in use. Haven't optimized old > > > indexes. We're pretty much just running ELK out of the box. When you > > > mention 'optimizing indexes', does this process combine indexes? > > > > No, but it can combine segments in a Lucene index (that make up > > Elasticsearch indexes), and segments are what's being backed up. > > So the more segments you have the the longer time snapshots are > > going to take. > > > > > Do you know if these performance problems are typical when > > > using ELK out of the box? > > > > 100 indexes on a single box should be okay but it depends on > > the size of the JVM heap. > > > > -- > > Magnus Bäck | Software Engineer, Development Tools > > [magnu...@sonymobile.com](mailto:magnu...@sonymobile.com) | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Aaron\_Mefford](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@Aaron\_Mefford](https://discuss.elastic.co/u/Aaron_Mefford)\ **Post date:** [March 13, 2015, 4:52pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/8 "2015-03-13T16:52:45Z")
Yes it was m1.smalls that I first noticed the EBS throttling on. Things work well in bursts, but sustained EBS does not work well. It will work substantially better in an m3.medium and if you are using the new EBS SSD volumes. On Thu, Mar 12, 2015 at 10:30 PM, Andy Nemzek [bitknower@gmail.com](mailto:bitknower@gmail.com) wrote: > Thank you guys for your thoughts here. This is really useful > information. Again, we're creating daily indexes because that's what > logstash does out of the box with the elasticsearch plugin, and this kind > of tuning info isn't included with that plugin. > > Minimizing both the number of indexes and shards now sounds like a great > idea. > > We are indeed using EC2. We're just using an m1.small that's EBS backed > (non-SSD). So, yes, it's not a very powerful machine, but again, we're not > throwing a lot of data at it either. > > On Thursday, March 12, 2015 at 12:50:22 PM UTC-5, [aa...@definemg.com](mailto:aa...@definemg.com) > wrote: > > > With the low volume of ingest, and the long duration of history, Id > > suggest you may want to trim back the number of shards per index from the > > default 5. Based on your 100 docs per day Id say 1 shard per day. If you > > combined this with the other suggestion to increase the duration of an > > index, then you might increase the number of shards, but maybe still not. > > Running an optimize once you have completed a time period is great advice > > if you can afford the overhead, sounds like one day at a time you should be > > able to, and that the overhead of not optimizing is costing you more when > > you snapshot. > > > > And index is made of shards, a shard is made of lucene segments. Lucene > > segments are the actual files that you copy when you snapshot. As such the > > number of segments is multiplied by the number of shards per index and the > > number of indexes. Reducing the number of indexes by creating larger time > > periods will significantly reduce the number of segments. Reducing the > > number of shards per index will significantly reduce the number of > > segments. Optimizing the index will also consolidate many segments into a > > single segment. > > > > Based on the use of S3 should we assume you are using AWS EC2? What > > instance size? Your data volume seems very low so it seems concerning that > > you have such a large time period to snapshot, and points to a slow file > > system, or a significant number of segments (100 indexes, 5 shards per > > index, xx segments per shard, == many thousands of segments). What does > > your storage system look like? If you are using EC2 are you using the > > newer EBS volumes (SSD backed)? Some of the smaller instance size > > significantly limit prolonged EBS throughput, in my experience. > > > > On Wednesday, March 11, 2015 at 1:12:01 AM UTC-6, Magnus Bäck wrote: > > > > > On Monday, March 09, 2015 at 20:29 CET, > > > Andy Nemzek [bitk...@gmail.com](mailto:bitk...@gmail.com) wrote: > > > > > > > We've been using logstash for several months now and it creates a new > > > > index each day, so I imagine there are over 100 indexes at this point. > > > > > > Why create daily indexes if you only have a few hundred entries in each? > > > There's a constant overhead for each shard so you don't want more > > > indexes than you need. Seems like you'd be fine with montly indexes, > > > and then your snapshot problems would disappear too. > > > > > > > Elasticsearch is running on a single machine...I haven't done anything > > > > with shards, so the defaults must be in use. Haven't optimized old > > > > indexes. We're pretty much just running ELK out of the box. When you > > > > mention 'optimizing indexes', does this process combine indexes? > > > > > > No, but it can combine segments in a Lucene index (that make up > > > Elasticsearch indexes), and segments are what's being backed up. > > > So the more segments you have the the longer time snapshots are > > > going to take. > > > > > > > Do you know if these performance problems are typical when > > > > using ELK out of the box? > > > > > > 100 indexes on a single box should be okay but it depends on > > > the size of the JVM heap. > > > > > > -- > > > Magnus Bäck | Software Engineer, Development Tools > > > [magnu...@sonymobile.com](mailto:magnu...@sonymobile.com) | Sony Mobile Communications > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe](https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAF9vEEpKGBV\_\_DJb8CfoN%3DvHZRB8zgYq002Bed%3DpQ-pVWXp%2BgQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF9vEEpKGBV__DJb8CfoN%3DvHZRB8zgYq002Bed%3DpQ-pVWXp%2BgQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andy\_Nemzek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_nemzek/32/776_2.png) [@Andy\_Nemzek](https://discuss.elastic.co/u/Andy_Nemzek)\ **Post date:** [March 16, 2015, 2:58pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/9 "2015-03-16T14:58:48Z")
Good to know, thanks for the tip! PS: per the prior conversations, I tried optimizing all the indexes so that they only contain 1 segment per shard. The operation actually did not take long, but it appears to have had only a marginal effect on the performance of snapshots. I'm not sure if it matters that the indexes are optimized, but the indexes in the old snapshots aren't...that is, will snapshotting get faster after those old snapshots have all been deleted and replaced with snapshots of the new optimized indexes? I guess the other two levers to try cranking are reducing shards and reducing indexes. If I understand correctly, there's no way to do this without writing up some script? On Friday, March 13, 2015 at 11:52:59 AM UTC-5, Aaron Mefford wrote: > Yes it was m1.smalls that I first noticed the EBS throttling on. Things > work well in bursts, but sustained EBS does not work well. It will work > substantially better in an m3.medium and if you are using the new EBS SSD > volumes. > > On Thu, Mar 12, 2015 at 10:30 PM, Andy Nemzek \<[bitk...@gmail.com](mailto:bitk...@gmail.com) > \\> wrote: > > > Thank you guys for your thoughts here. This is really useful > > information. Again, we're creating daily indexes because that's what > > logstash does out of the box with the elasticsearch plugin, and this kind > > of tuning info isn't included with that plugin. > > > > Minimizing both the number of indexes and shards now sounds like a great > > idea. > > > > We are indeed using EC2. We're just using an m1.small that's EBS backed > > (non-SSD). So, yes, it's not a very powerful machine, but again, we're not > > throwing a lot of data at it either. > > > > On Thursday, March 12, 2015 at 12:50:22 PM UTC-5, [aa...@definemg.com](mailto:aa...@definemg.com) > > wrote: > > > > > With the low volume of ingest, and the long duration of history, Id > > > suggest you may want to trim back the number of shards per index from the > > > default 5. Based on your 100 docs per day Id say 1 shard per day. If you > > > combined this with the other suggestion to increase the duration of an > > > index, then you might increase the number of shards, but maybe still not. > > > Running an optimize once you have completed a time period is great advice > > > if you can afford the overhead, sounds like one day at a time you should be > > > able to, and that the overhead of not optimizing is costing you more when > > > you snapshot. > > > > > > And index is made of shards, a shard is made of lucene segments. Lucene > > > segments are the actual files that you copy when you snapshot. As such the > > > number of segments is multiplied by the number of shards per index and the > > > number of indexes. Reducing the number of indexes by creating larger time > > > periods will significantly reduce the number of segments. Reducing the > > > number of shards per index will significantly reduce the number of > > > segments. Optimizing the index will also consolidate many segments into a > > > single segment. > > > > > > Based on the use of S3 should we assume you are using AWS EC2? What > > > instance size? Your data volume seems very low so it seems concerning that > > > you have such a large time period to snapshot, and points to a slow file > > > system, or a significant number of segments (100 indexes, 5 shards per > > > index, xx segments per shard, == many thousands of segments). What does > > > your storage system look like? If you are using EC2 are you using the > > > newer EBS volumes (SSD backed)? Some of the smaller instance size > > > significantly limit prolonged EBS throughput, in my experience. > > > > > > On Wednesday, March 11, 2015 at 1:12:01 AM UTC-6, Magnus Bäck wrote: > > > > > > > On Monday, March 09, 2015 at 20:29 CET, > > > > Andy Nemzek [bitk...@gmail.com](mailto:bitk...@gmail.com) wrote: > > > > > > > > > We've been using logstash for several months now and it creates a new > > > > > index each day, so I imagine there are over 100 indexes at this > > > > > point. > > > > > > > > Why create daily indexes if you only have a few hundred entries in > > > > each? > > > > There's a constant overhead for each shard so you don't want more > > > > indexes than you need. Seems like you'd be fine with montly indexes, > > > > and then your snapshot problems would disappear too. > > > > > > > > > Elasticsearch is running on a single machine...I haven't done > > > > > anything > > > > > with shards, so the defaults must be in use. Haven't optimized old > > > > > indexes. We're pretty much just running ELK out of the box. When > > > > > you > > > > > mention 'optimizing indexes', does this process combine indexes? > > > > > > > > No, but it can combine segments in a Lucene index (that make up > > > > Elasticsearch indexes), and segments are what's being backed up. > > > > So the more segments you have the the longer time snapshots are > > > > going to take. > > > > > > > > > Do you know if these performance problems are typical when > > > > > using ELK out of the box? > > > > > > > > 100 indexes on a single box should be okay but it depends on > > > > the size of the JVM heap. > > > > > > > > -- > > > > Magnus Bäck | Software Engineer, Development Tools > > > > [magnu...@sonymobile.com](mailto:magnu...@sonymobile.com) | Sony Mobile Communications > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe](https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Aaron\_Mefford](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@Aaron\_Mefford](https://discuss.elastic.co/u/Aaron_Mefford)\ **Post date:** [March 16, 2015, 5:45pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/10 "2015-03-16T17:45:32Z")
I don't know precisely the snapshoting procedure that you are using, but in general, yes if you have not optimized the old indexes, then you likely have many more shards/indexes in old at this point than you do optimized and they should age off or you can start tackling them one by one. Regarding the reducing of shards and indexes. In order to merge existing indexes, yes, a script or possibly a plugin would be required. There is nothing built-in today to accomplish this task. I think there are features coming that will make this easier in 1.5 or 2.0. Honestly though, I think your biggest issue is an undersized instance. If you need to stay at that size, I can understand that, but you need to make sure you are effectively evaluating the cost of your engineering efforts vs the cost of an increase in instance size. You could spend a lot of hours trying to make something work, with little success, which would just work, with a larger instance. It is also possible that you are exceeding the IOPS for an m1.small or your EBS volume. Upgrading will improve your IO, reducing the number of segments will reduce the need for IO. You may want to consider moving to a t2.medium which is in the same ballpark of cost, but can burst to much better performance. On Mon, Mar 16, 2015 at 8:58 AM, Andy Nemzek [bitknower@gmail.com](mailto:bitknower@gmail.com) wrote: > Good to know, thanks for the tip! > > PS: per the prior conversations, I tried optimizing all the indexes so > that they only contain 1 segment per shard. The operation actually did not > take long, but it appears to have had only a marginal effect on the > performance of snapshots. I'm not sure if it matters that the indexes are > optimized, but the indexes in the old snapshots aren't...that is, will > snapshotting get faster after those old snapshots have all been deleted and > replaced with snapshots of the new optimized indexes? > > I guess the other two levers to try cranking are reducing shards and > reducing indexes. If I understand correctly, there's no way to do this > without writing up some script? > > On Friday, March 13, 2015 at 11:52:59 AM UTC-5, Aaron Mefford wrote: > > > Yes it was m1.smalls that I first noticed the EBS throttling on. Things > > work well in bursts, but sustained EBS does not work well. It will work > > substantially better in an m3.medium and if you are using the new EBS SSD > > volumes. > > > > On Thu, Mar 12, 2015 at 10:30 PM, Andy Nemzek [bitk...@gmail.com](mailto:bitk...@gmail.com) wrote: > > > > > Thank you guys for your thoughts here. This is really useful > > > information. Again, we're creating daily indexes because that's what > > > logstash does out of the box with the elasticsearch plugin, and this kind > > > of tuning info isn't included with that plugin. > > > > > > Minimizing both the number of indexes and shards now sounds like a great > > > idea. > > > > > > We are indeed using EC2. We're just using an m1.small that's EBS backed > > > (non-SSD). So, yes, it's not a very powerful machine, but again, we're not > > > throwing a lot of data at it either. > > > > > > On Thursday, March 12, 2015 at 12:50:22 PM UTC-5, [aa...@definemg.com](mailto:aa...@definemg.com) > > > wrote: > > > > > > > With the low volume of ingest, and the long duration of history, Id > > > > suggest you may want to trim back the number of shards per index from the > > > > default 5. Based on your 100 docs per day Id say 1 shard per day. If you > > > > combined this with the other suggestion to increase the duration of an > > > > index, then you might increase the number of shards, but maybe still not. > > > > Running an optimize once you have completed a time period is great advice > > > > if you can afford the overhead, sounds like one day at a time you should be > > > > able to, and that the overhead of not optimizing is costing you more when > > > > you snapshot. > > > > > > > > And index is made of shards, a shard is made of lucene segments. > > > > Lucene segments are the actual files that you copy when you snapshot. As > > > > such the number of segments is multiplied by the number of shards per index > > > > and the number of indexes. Reducing the number of indexes by creating > > > > larger time periods will significantly reduce the number of segments. > > > > Reducing the number of shards per index will significantly reduce the > > > > number of segments. Optimizing the index will also consolidate many > > > > segments into a single segment. > > > > > > > > Based on the use of S3 should we assume you are using AWS EC2? What > > > > instance size? Your data volume seems very low so it seems concerning that > > > > you have such a large time period to snapshot, and points to a slow file > > > > system, or a significant number of segments (100 indexes, 5 shards per > > > > index, xx segments per shard, == many thousands of segments). What does > > > > your storage system look like? If you are using EC2 are you using the > > > > newer EBS volumes (SSD backed)? Some of the smaller instance size > > > > significantly limit prolonged EBS throughput, in my experience. > > > > > > > > On Wednesday, March 11, 2015 at 1:12:01 AM UTC-6, Magnus Bäck wrote: > > > > > > > > > On Monday, March 09, 2015 at 20:29 CET, > > > > > Andy Nemzek [bitk...@gmail.com](mailto:bitk...@gmail.com) wrote: > > > > > > > > > > > We've been using logstash for several months now and it creates a > > > > > > new > > > > > > index each day, so I imagine there are over 100 indexes at this > > > > > > point. > > > > > > > > > > Why create daily indexes if you only have a few hundred entries in > > > > > each? > > > > > There's a constant overhead for each shard so you don't want more > > > > > indexes than you need. Seems like you'd be fine with montly indexes, > > > > > and then your snapshot problems would disappear too. > > > > > > > > > > > Elasticsearch is running on a single machine...I haven't done > > > > > > anything > > > > > > with shards, so the defaults must be in use. Haven't optimized old > > > > > > indexes. We're pretty much just running ELK out of the box. When > > > > > > you > > > > > > mention 'optimizing indexes', does this process combine indexes? > > > > > > > > > > No, but it can combine segments in a Lucene index (that make up > > > > > Elasticsearch indexes), and segments are what's being backed up. > > > > > So the more segments you have the the longer time snapshots are > > > > > going to take. > > > > > > > > > > > Do you know if these performance problems are typical when > > > > > > using ELK out of the box? > > > > > > > > > > 100 indexes on a single box should be okay but it depends on > > > > > the size of the JVM heap. > > > > > > > > > > -- > > > > > Magnus Bäck | Software Engineer, Development Tools > > > > > [magnu...@sonymobile.com](mailto:magnu...@sonymobile.com) | Sony Mobile Communications > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > topic/elasticsearch/VEdqtEpc3ac/unsubscribe. > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe](https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAF9vEEr8ORchWkznq4MvuuL3%2ByNV6kkDX\_z0QOeaD3%2BkPUwTjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF9vEEr8ORchWkznq4MvuuL3%2ByNV6kkDX_z0QOeaD3%2BkPUwTjQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andy\_Nemzek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_nemzek/32/776_2.png) [@Andy\_Nemzek](https://discuss.elastic.co/u/Andy_Nemzek)\ **Post date:** [April 9, 2015, 6:59pm UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/11 "2015-04-09T18:59:23Z")
I wanted to follow up here. So we tried snapshotting to disk instead of using the s3 plugin. We've been running this way for the better part of a month and the snapshot/cleanup process appears to only take a couple minutes now...down from over an hour. So the problems we were experiencing were apparently due to the plugin and how it manipulates data on s3. We plan on snapshotting to disk and simply syncing that to s3 for permanent storage. Hopefully this will help others with the same problem! On Monday, March 16, 2015 at 12:45:48 PM UTC-5, Aaron Mefford wrote: > I don't know precisely the snapshoting procedure that you are using, but > in general, yes if you have not optimized the old indexes, then you likely > have many more shards/indexes in old at this point than you do optimized > and they should age off or you can start tackling them one by one. > > Regarding the reducing of shards and indexes. In order to merge existing > indexes, yes, a script or possibly a plugin would be required. There is > nothing built-in today to accomplish this task. I think there are features > coming that will make this easier in 1.5 or 2.0. Honestly though, I think > your biggest issue is an undersized instance. If you need to stay at that > size, I can understand that, but you need to make sure you are effectively > evaluating the cost of your engineering efforts vs the cost of an increase > in instance size. You could spend a lot of hours trying to make something > work, with little success, which would just work, with a larger instance. > > It is also possible that you are exceeding the IOPS for an m1.small or > your EBS volume. Upgrading will improve your IO, reducing the number of > segments will reduce the need for IO. You may want to consider moving to a > t2.medium which is in the same ballpark of cost, but can burst to much > better performance. > > On Mon, Mar 16, 2015 at 8:58 AM, Andy Nemzek \<[bitk...@gmail.com](mailto:bitk...@gmail.com) > \\> wrote: > > > Good to know, thanks for the tip! > > > > PS: per the prior conversations, I tried optimizing all the indexes so > > that they only contain 1 segment per shard. The operation actually did not > > take long, but it appears to have had only a marginal effect on the > > performance of snapshots. I'm not sure if it matters that the indexes are > > optimized, but the indexes in the old snapshots aren't...that is, will > > snapshotting get faster after those old snapshots have all been deleted and > > replaced with snapshots of the new optimized indexes? > > > > I guess the other two levers to try cranking are reducing shards and > > reducing indexes. If I understand correctly, there's no way to do this > > without writing up some script? > > > > On Friday, March 13, 2015 at 11:52:59 AM UTC-5, Aaron Mefford wrote: > > > > > Yes it was m1.smalls that I first noticed the EBS throttling on. Things > > > work well in bursts, but sustained EBS does not work well. It will work > > > substantially better in an m3.medium and if you are using the new EBS SSD > > > volumes. > > > > > > On Thu, Mar 12, 2015 at 10:30 PM, Andy Nemzek [bitk...@gmail.com](mailto:bitk...@gmail.com) wrote: > > > > > > > Thank you guys for your thoughts here. This is really useful > > > > information. Again, we're creating daily indexes because that's what > > > > logstash does out of the box with the elasticsearch plugin, and this kind > > > > of tuning info isn't included with that plugin. > > > > > > > > Minimizing both the number of indexes and shards now sounds like a > > > > great idea. > > > > > > > > We are indeed using EC2. We're just using an m1.small that's EBS > > > > backed (non-SSD). So, yes, it's not a very powerful machine, but again, > > > > we're not throwing a lot of data at it either. > > > > > > > > On Thursday, March 12, 2015 at 12:50:22 PM UTC-5, [aa...@definemg.com](mailto:aa...@definemg.com) > > > > wrote: > > > > > > > > > With the low volume of ingest, and the long duration of history, Id > > > > > suggest you may want to trim back the number of shards per index from the > > > > > default 5. Based on your 100 docs per day Id say 1 shard per day. If you > > > > > combined this with the other suggestion to increase the duration of an > > > > > index, then you might increase the number of shards, but maybe still not. > > > > > Running an optimize once you have completed a time period is great advice > > > > > if you can afford the overhead, sounds like one day at a time you should be > > > > > able to, and that the overhead of not optimizing is costing you more when > > > > > you snapshot. > > > > > > > > > > And index is made of shards, a shard is made of lucene segments. > > > > > Lucene segments are the actual files that you copy when you snapshot. As > > > > > such the number of segments is multiplied by the number of shards per index > > > > > and the number of indexes. Reducing the number of indexes by creating > > > > > larger time periods will significantly reduce the number of segments. > > > > > Reducing the number of shards per index will significantly reduce the > > > > > number of segments. Optimizing the index will also consolidate many > > > > > segments into a single segment. > > > > > > > > > > Based on the use of S3 should we assume you are using AWS EC2? What > > > > > instance size? Your data volume seems very low so it seems concerning that > > > > > you have such a large time period to snapshot, and points to a slow file > > > > > system, or a significant number of segments (100 indexes, 5 shards per > > > > > index, xx segments per shard, == many thousands of segments). What does > > > > > your storage system look like? If you are using EC2 are you using the > > > > > newer EBS volumes (SSD backed)? Some of the smaller instance size > > > > > significantly limit prolonged EBS throughput, in my experience. > > > > > > > > > > On Wednesday, March 11, 2015 at 1:12:01 AM UTC-6, Magnus Bäck wrote: > > > > > > > > > > > On Monday, March 09, 2015 at 20:29 CET, > > > > > > Andy Nemzek [bitk...@gmail.com](mailto:bitk...@gmail.com) wrote: > > > > > > > > > > > > > We've been using logstash for several months now and it creates a > > > > > > > new > > > > > > > index each day, so I imagine there are over 100 indexes at this > > > > > > > point. > > > > > > > > > > > > Why create daily indexes if you only have a few hundred entries in > > > > > > each? > > > > > > There's a constant overhead for each shard so you don't want more > > > > > > indexes than you need. Seems like you'd be fine with montly indexes, > > > > > > and then your snapshot problems would disappear too. > > > > > > > > > > > > > Elasticsearch is running on a single machine...I haven't done > > > > > > > anything > > > > > > > with shards, so the defaults must be in use. Haven't optimized old > > > > > > > indexes. We're pretty much just running ELK out of the box. When > > > > > > > you > > > > > > > mention 'optimizing indexes', does this process combine indexes? > > > > > > > > > > > > No, but it can combine segments in a Lucene index (that make up > > > > > > Elasticsearch indexes), and segments are what's being backed up. > > > > > > So the more segments you have the the longer time snapshots are > > > > > > going to take. > > > > > > > > > > > > > Do you know if these performance problems are typical when > > > > > > > using ELK out of the box? > > > > > > > > > > > > 100 indexes on a single box should be okay but it depends on > > > > > > the size of the JVM heap. > > > > > > > > > > > > -- > > > > > > Magnus Bäck | Software Engineer, Development Tools > > > > > > [magnu...@sonymobile.com](mailto:magnu...@sonymobile.com) | Sony Mobile Communications > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > topic/elasticsearch/VEdqtEpc3ac/unsubscribe. > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/58a94899-185c-4c4d-ad5f-ac2e0a5eed2d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe](https://groups.google.com/d/topic/elasticsearch/VEdqtEpc3ac/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/13ae8aeb-6e07-4a00-bc2a-f8792adc4d9d%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3ff4db78-3bc5-46a8-8802-ab3f9631e9c1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3ff4db78-3bc5-46a8-8802-ab3f9631e9c1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:20am UTC](https://discuss.elastic.co/t/snapshot-scaling-problems/22586/12 "2017-07-06T00:20:33Z")