# Snapshot Duration increasing over time **URL:** \ **Category:** Elasticsearch\ **Created:** [May 7, 2014, 10:58am UTC](https://discuss.elastic.co/t/snapshot-duration-increasing-over-time/17389 "2014-05-07T10:58:05Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Dipesh\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dipesh_patel/32/1579_2.png) [@Dipesh\_Patel](https://discuss.elastic.co/u/Dipesh_Patel)\ **Post date:** [May 7, 2014, 10:58am UTC](https://discuss.elastic.co/t/snapshot-duration-increasing-over-time/17389/1 "2014-05-07T10:58:05Z")
Hi We've noticed recently that our snapshot durations are increasing over time. Our rate of flow of data going into elasticsearch has remained fairly constant. Though we do create new indices everyday ( though this is a fixed number that doesn't vary from day to day ). We are currently snapshoting, or trying to snapshot every hour. However with the snapshots taking a progressively longer time this is proving difficult. Here's some stats showing our time to finish: Name Duration ( milli ) snapshot\_2014-05-01\_01:30:00 4497010 snapshot\_2014-05-01\_03:30:00 4513037 snapshot\_2014-05-01\_05:30:00 4770288 snapshot\_2014-05-01\_07:30:00 5413361 snapshot\_2014-05-01\_11:30:00 6978384 snapshot\_2014-05-01\_13:30:00 6907554 snapshot\_2014-05-01\_15:30:00 7388500 This is just the tail end originally the snapshots were only taking 7-8 mins to run, they've just been getting progressively longer. Any help to debug etc appreciated. Dip -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ff3a7794-80c2-4c22-9467-18274e0308ae%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ff3a7794-80c2-4c22-9467-18274e0308ae%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [May 8, 2014, 1:58pm UTC](https://discuss.elastic.co/t/snapshot-duration-increasing-over-time/17389/2 "2014-05-08T13:58:22Z")
Hi Dipesh, I have a few questions. Are you still on S3? Which version of elasticsearch are you using? How many snapshots do you currently keep in S3? How fast is your index growing over time? Igor On Wednesday, May 7, 2014 6:58:05 AM UTC-4, Dipesh Patel wrote: > Hi > > We've noticed recently that our snapshot durations are increasing over > time. Our rate of flow of data going into elasticsearch has remained fairly > constant. Though we do create new indices everyday ( though this is a fixed > number that doesn't vary from day to day ). We are currently snapshoting, > or trying to snapshot every hour. However with the snapshots taking a > progressively longer time this is proving difficult. > > Here's some stats showing our time to finish: > > Name > Duration ( milli ) > snapshot\_2014-05-01\_01:30:00 4497010 snapshot\_2014-05-01\_03:30:00 > 4513037 snapshot\_2014-05-01\_05:30:00 4770288 > snapshot\_2014-05-01\_07:30:00 5413361 snapshot\_2014-05-01\_11:30:00 6978384 > snapshot\_2014-05-01\_13:30:00 6907554 snapshot\_2014-05-01\_15:30:00 7388500 > This is just the tail end originally the snapshots were only taking 7-8 > mins to run, they've just been getting progressively longer. > > Any help to debug etc appreciated. > > Dip -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/96ca5fa6-f366-4f9e-b3c1-a57b511a341e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/96ca5fa6-f366-4f9e-b3c1-a57b511a341e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Dipesh\_Patel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dipesh_patel/32/1579_2.png) [@Dipesh\_Patel](https://discuss.elastic.co/u/Dipesh_Patel)\ **Post date:** [May 8, 2014, 2:17pm UTC](https://discuss.elastic.co/t/snapshot-duration-increasing-over-time/17389/3 "2014-05-08T14:17:37Z")
Hi Igor We are using elasticsearch 1.1.1. Currently we are keeping all snapshots that we make in s3, we haven't yet decided on an archive strategy/solution. So at the moment we have 131 snapshots in the s3 bucket. So we have about 112 new indices a day. I'll explain our set up a bit it may well be that we should be doing something different. We are grabbing application logs from lots of different apps and putting them into elasticsearch. We are using flume to do this. So similar to logstash setup. The one major difference is that we are creating an index for each application. So if we have 100 apps we will create 100 indices each day one for each new app. Dip On Thursday, May 8, 2014 2:58:22 PM UTC+1, Igor Motov wrote: > Hi Dipesh, > > I have a few questions. Are you still on S3? Which version of > elasticsearch are you using? How many snapshots do you currently keep in > S3? How fast is your index growing over time? > > Igor > > On Wednesday, May 7, 2014 6:58:05 AM UTC-4, Dipesh Patel wrote: > > > Hi > > > > We've noticed recently that our snapshot durations are increasing over > > time. Our rate of flow of data going into elasticsearch has remained fairly > > constant. Though we do create new indices everyday ( though this is a fixed > > number that doesn't vary from day to day ). We are currently snapshoting, > > or trying to snapshot every hour. However with the snapshots taking a > > progressively longer time this is proving difficult. > > > > Here's some stats showing our time to finish: > > > > Name > > Duration ( milli ) > > snapshot\_2014-05-01\_01:30:00 4497010 snapshot\_2014-05-01\_03:30:00 > > 4513037 snapshot\_2014-05-01\_05:30:00 4770288 > > snapshot\_2014-05-01\_07:30:00 5413361 snapshot\_2014-05-01\_11:30:00 > > 6978384 snapshot\_2014-05-01\_13:30:00 6907554 > > snapshot\_2014-05-01\_15:30:00 7388500 > > This is just the tail end originally the snapshots were only taking 7-8 > > mins to run, they've just been getting progressively longer. > > > > Any help to debug etc appreciated. > > > > Dip -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/441319d1-a521-4a6b-9671-de541d129029%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/441319d1-a521-4a6b-9671-de541d129029%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Sally\_Ahn](https://avatars.discourse-cdn.com/v4/letter/s/ba8739/32.png) [@Sally\_Ahn](https://discuss.elastic.co/u/Sally_Ahn)\ **Post date:** [November 13, 2014, 7:04am UTC](https://discuss.elastic.co/t/snapshot-duration-increasing-over-time/17389/4 "2014-11-13T07:04:42Z")
Hello, I am also running into this issue. I've specified more details in this thread [https://groups.google.com/d/msg/elasticsearch/cXl9UrqgwBI/et56xPUqwGkJ](https://groups.google.com/d/msg/elasticsearch/cXl9UrqgwBI/et56xPUqwGkJ). I would greatly appreciate any response. Thanks, Sally On Thursday, May 8, 2014 7:17:37 AM UTC-7, Dipesh Patel wrote: > Hi Igor > > We are using elasticsearch 1.1.1. > Currently we are keeping all snapshots that we make in s3, we haven't yet > decided on an archive strategy/solution. So at the moment we have 131 > snapshots in the s3 bucket. > So we have about 112 new indices a day. > > I'll explain our set up a bit it may well be that we should be doing > something different. We are grabbing application logs from lots of > different apps and putting them into elasticsearch. We are using flume to > do this. So similar to logstash setup. The one major difference is that we > are creating an index for each application. So if we have 100 apps we will > create 100 indices each day one for each new app. > > Dip > > On Thursday, May 8, 2014 2:58:22 PM UTC+1, Igor Motov wrote: > > > Hi Dipesh, > > > > I have a few questions. Are you still on S3? Which version of > > elasticsearch are you using? How many snapshots do you currently keep in > > S3? How fast is your index growing over time? > > > > Igor > > > > On Wednesday, May 7, 2014 6:58:05 AM UTC-4, Dipesh Patel wrote: > > > > > Hi > > > > > > We've noticed recently that our snapshot durations are increasing over > > > time. Our rate of flow of data going into elasticsearch has remained fairly > > > constant. Though we do create new indices everyday ( though this is a fixed > > > number that doesn't vary from day to day ). We are currently snapshoting, > > > or trying to snapshot every hour. However with the snapshots taking a > > > progressively longer time this is proving difficult. > > > > > > Here's some stats showing our time to finish: > > > > > > Name > > > Duration ( milli ) > > > snapshot\_2014-05-01\_01:30:00 4497010 snapshot\_2014-05-01\_03:30:00 > > > 4513037 snapshot\_2014-05-01\_05:30:00 4770288 > > > snapshot\_2014-05-01\_07:30:00 5413361 snapshot\_2014-05-01\_11:30:00 > > > 6978384 snapshot\_2014-05-01\_13:30:00 6907554 > > > snapshot\_2014-05-01\_15:30:00 7388500 > > > This is just the tail end originally the snapshots were only taking 7-8 > > > mins to run, they've just been getting progressively longer. > > > > > > Any help to debug etc appreciated. > > > > > > Dip -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cc065c1c-3d30-4252-8696-0a51740a24b8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cc065c1c-3d30-4252-8696-0a51740a24b8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [December 3, 2014, 12:26pm UTC](https://discuss.elastic.co/t/snapshot-duration-increasing-over-time/17389/5 "2014-12-03T12:26:47Z")
Snapshots are at the segment level. The more segments stored in the repository, the more segments will have to be compared to those in each successive snapshot. With merges taking place continually in an active index, you may end up with a considerable number of "orphaned" segments stored in your repository, i.e. segments "backed up," but no longer directly correlating to a segment in your index. Checking through these may be contributing to the increased amount of time between snapshots. Consider pruning older snapshots. "Orphaned" segments will be deleted, and any segments still referenced will be preserved. --Aaron On Thursday, November 13, 2014 2:04:42 AM UTC-5, Sally Ahn wrote: > Hello, > > I am also running into this issue. I've specified more details in this > thread > [https://groups.google.com/d/msg/elasticsearch/cXl9UrqgwBI/et56xPUqwGkJ](https://groups.google.com/d/msg/elasticsearch/cXl9UrqgwBI/et56xPUqwGkJ). > I would greatly appreciate any response. > > Thanks, > Sally > > On Thursday, May 8, 2014 7:17:37 AM UTC-7, Dipesh Patel wrote: > > > Hi Igor > > > > We are using elasticsearch 1.1.1. > > Currently we are keeping all snapshots that we make in s3, we haven't yet > > decided on an archive strategy/solution. So at the moment we have 131 > > snapshots in the s3 bucket. > > So we have about 112 new indices a day. > > > > I'll explain our set up a bit it may well be that we should be doing > > something different. We are grabbing application logs from lots of > > different apps and putting them into elasticsearch. We are using flume to > > do this. So similar to logstash setup. The one major difference is that we > > are creating an index for each application. So if we have 100 apps we will > > create 100 indices each day one for each new app. > > > > Dip > > > > On Thursday, May 8, 2014 2:58:22 PM UTC+1, Igor Motov wrote: > > > > > Hi Dipesh, > > > > > > I have a few questions. Are you still on S3? Which version of > > > elasticsearch are you using? How many snapshots do you currently keep in > > > S3? How fast is your index growing over time? > > > > > > Igor > > > > > > On Wednesday, May 7, 2014 6:58:05 AM UTC-4, Dipesh Patel wrote: > > > > > > > Hi > > > > > > > > We've noticed recently that our snapshot durations are increasing over > > > > time. Our rate of flow of data going into elasticsearch has remained fairly > > > > constant. Though we do create new indices everyday ( though this is a fixed > > > > number that doesn't vary from day to day ). We are currently snapshoting, > > > > or trying to snapshot every hour. However with the snapshots taking a > > > > progressively longer time this is proving difficult. > > > > > > > > Here's some stats showing our time to finish: > > > > > > > > Name > > > > Duration ( milli ) > > > > snapshot\_2014-05-01\_01:30:00 4497010 snapshot\_2014-05-01\_03:30:00 > > > > 4513037 snapshot\_2014-05-01\_05:30:00 4770288 > > > > snapshot\_2014-05-01\_07:30:00 5413361 snapshot\_2014-05-01\_11:30:00 > > > > 6978384 snapshot\_2014-05-01\_13:30:00 6907554 > > > > snapshot\_2014-05-01\_15:30:00 7388500 > > > > This is just the tail end originally the snapshots were only taking 7-8 > > > > mins to run, they've just been getting progressively longer. > > > > > > > > Any help to debug etc appreciated. > > > > > > > > Dip -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a2eccb59-10fa-4a63-80b0-c8a0880cb068%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a2eccb59-10fa-4a63-80b0-c8a0880cb068%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:46am UTC](https://discuss.elastic.co/t/snapshot-duration-increasing-over-time/17389/6 "2017-07-06T00:46:18Z")