# Accumulate a ttl index into a large backup index
**URL:** https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635
**Category:** Elasticsearch
**Created:** [January 14, 2015, 1:01pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635 "2015-01-14T13:01:46Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Behrad\_Zari](https://discuss.elastic.co/u/Behrad_Zari)
#### Post date: [January 14, 2015, 1:01pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/1 "2015-01-14T13:01:46Z")
My data contains a weekly ttl so my ES index will loose older data, however
I want to have another accumulative large index containing a year of my
data... How can I instruct ES/sync my second index without affecting ttls
in the first index? ( I don't want deleted docs from first index to be
deleted in the second)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 14, 2015, 1:13pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/2 "2015-01-14T13:13:01Z")
Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.
So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)
HTH
David
> Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> ## My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)
>
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Behrad\_Zari](https://discuss.elastic.co/u/Behrad_Zari)
#### Post date: [January 14, 2015, 1:21pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/3 "2015-01-14T13:21:46Z")
I am actually using Couchbase ES plugin David, this is my database
requirement that data has a weekly ttl, so when data is removed in
Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
2015-01-14 16:43 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> Don't use ttl for that kind of usage but prefer time based indices.
> Much more efficient.
>
> So the year index won't have any removal and you will end up to just
> remove weekly indices which is super fast and doesn't generate a lot of IO
> (but ttl does)
>
> HTH
>
> David
>
> Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> My data contains a weekly ttl so my ES index will loose older data,
> however I want to have another accumulative large index containing a year
> of my data... How can I instruct ES/sync my second index without affecting
> ttls in the first index? ( I don't want deleted docs from first index to be
> deleted in the second)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr)
> [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
--Behrad
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 14, 2015, 1:30pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/4 "2015-01-14T13:30:36Z")
May be you could use different mapping for the year index and the week index?
One with ttl disabled and the other one with ttl enabled?
## [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html) [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
@dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> Le 14 janv. 2015 à 14:21, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
>
> 2015-01-14 16:43 GMT+03:30 David Pilato \<[david@pilato.fr](mailto:david@pilato.fr) [mailto:david@pilato.fr](mailto:david@pilato.fr)\>:
> Don't use ttl for that kind of usage but prefer time based indices.
> Much more efficient.
>
> So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)
>
> HTH
>
> David
>
> Le 14 janv. 2015 à 14:01, Behrad Zari \<[behradz@gmail.com](mailto:behradz@gmail.com) [mailto:behradz@gmail.com](mailto:behradz@gmail.com)\> a écrit :
>
> > My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)
> >
> > --
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer).
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe) [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr) [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer).
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> --Behrad
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Behrad\_Zari](https://discuss.elastic.co/u/Behrad_Zari)
#### Post date: [January 14, 2015, 1:51pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/5 "2015-01-14T13:51:09Z")
I'll recheck elasticsearch-transport-couchbase again David, another two
related questions comes to my mind to ask,
1. Is there any best practice to replicate/sync/duplicate an existing index
to a new index by ES?
2. Considering I have an existing index with data persist enabled on ES,
Can I use that to create a new index with a customer defined mapping on my
data on ES without the need of my primary database? (does sroll api help
with this?)
2015-01-14 17:00 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> May be you could use different mapping for the year index and the week
> index?
> One with ttl disabled and the other one with ttl enabled?
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
>
> --
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)
> [http://Elasticsearch.com](http://Elasticsearch.com)_
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr
> [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs
> [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
>
> Le 14 janv. 2015 à 14:21, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> I am actually using Couchbase ES plugin David, this is my database
> requirement that data has a weekly ttl, so when data is removed in
> Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
>
> 2015-01-14 16:43 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
>
> > Don't use ttl for that kind of usage but prefer time based indices.
> > Much more efficient.
> >
> > So the year index won't have any removal and you will end up to just
> > remove weekly indices which is super fast and doesn't generate a lot of IO
> > (but ttl does)
> >
> > HTH
> >
> > David
> >
> > Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> >
> > My data contains a weekly ttl so my ES index will loose older data,
> > however I want to have another accumulative large index containing a year
> > of my data... How can I instruct ES/sync my second index without affecting
> > ttls in the first index? ( I don't want deleted docs from first index to be
> > deleted in the second)
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit
> > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > To unsubscribe from this group and all its topics, send an email to
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr)
> > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer)
> > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> --Behrad
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr)
> [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
--Behrad
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 14, 2015, 2:55pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/6 "2015-01-14T14:55:58Z")
Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.
That said, reading from couchbase is something super easy. So I guess you could may be replicate your data from couchbase twice using 2 XDCR configs one for the weekly index and the other one for the year index. Is that something doable?
--
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
@dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> Le 14 janv. 2015 à 14:51, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> I'll recheck elasticsearch-transport-couchbase again David, another two related questions comes to my mind to ask,
>
> 1. Is there any best practice to replicate/sync/duplicate an existing index to a new index by ES?
> 2. Considering I have an existing index with data persist enabled on ES, Can I use that to create a new index with a customer defined mapping on my data on ES without the need of my primary database? (does sroll api help with this?)
>
> 2015-01-14 17:00 GMT+03:30 David Pilato \<[david@pilato.fr](mailto:david@pilato.fr) [mailto:david@pilato.fr](mailto:david@pilato.fr)\>:
> May be you could use different mapping for the year index and the week index?
> One with ttl disabled and the other one with ttl enabled?
>
> ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html) [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
>
> David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) [http://elasticsearch.com/](http://elasticsearch.com/)
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
>
> > Le 14 janv. 2015 à 14:21, Behrad \<[behradz@gmail.com](mailto:behradz@gmail.com) [mailto:behradz@gmail.com](mailto:behradz@gmail.com)\> a écrit :
> >
> > I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
> >
> > 2015-01-14 16:43 GMT+03:30 David Pilato \<[david@pilato.fr](mailto:david@pilato.fr) [mailto:david@pilato.fr](mailto:david@pilato.fr)\>:
> > Don't use ttl for that kind of usage but prefer time based indices.
> > Much more efficient.
> >
> > So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)
> >
> > HTH
> >
> > David
> >
> > Le 14 janv. 2015 à 14:01, Behrad Zari \<[behradz@gmail.com](mailto:behradz@gmail.com) [mailto:behradz@gmail.com](mailto:behradz@gmail.com)\> a écrit :
> >
> > > My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer).
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe) [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr) [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer).
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > --Behrad
> >
> > --
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer).
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe) [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr) [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer).
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> --Behrad
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Behrad\_Zari](https://discuss.elastic.co/u/Behrad_Zari)
#### Post date: [January 14, 2015, 5:18pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/7 "2015-01-14T17:18:56Z")
2015-01-14 18:25 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> Definitely scan and scroll API is for that.
> Some clients provide that feature out of the box.
>
> That said, reading from couchbase is something super easy. So I guess you
> could may be replicate your data from couchbase twice using 2 XDCR configs
> one for the weekly index and the other one for the year index. Is that
> something doable?
Unfortunately not David, XDCR blinding syncs the source db with remotes
(any deleted data will also be deleted from the remote) and we can't let
the source db to grow (about 1.3 billion records
/month) so we are using couchbase TTLs to store a week or so,
however we'd like to have online searching/access to a multiple month (i
hope by utilizing ES) and I'm seeking the best Enterprise approach to this.
However I haven't any clue for capacity planning of ES for 5, 6 billions of
records.
> --
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)
> [http://Elasticsearch.com](http://Elasticsearch.com)_
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr
> [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs
> [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
>
> Le 14 janv. 2015 à 14:51, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> I'll recheck elasticsearch-transport-couchbase again David, another two
> related questions comes to my mind to ask,
>
> 1. Is there any best practice to replicate/sync/duplicate an existing
> index to a new index by ES?
> 2. Considering I have an existing index with data persist enabled on ES,
> Can I use that to create a new index with a customer defined mapping on my
> data on ES without the need of my primary database? (does sroll api help
> with this?)
>
> 2015-01-14 17:00 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
>
> > May be you could use different mapping for the year index and the week
> > index?
> > One with ttl disabled and the other one with ttl enabled?
> >
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
> >
> > --
> > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)
> > [http://elasticsearch.com/](http://elasticsearch.com/)_
> > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr
> > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs
> > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> >
> > Le 14 janv. 2015 à 14:21, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> >
> > I am actually using Couchbase ES plugin David, this is my database
> > requirement that data has a weekly ttl, so when data is removed in
> > Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
> >
> > 2015-01-14 16:43 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> >
> > > Don't use ttl for that kind of usage but prefer time based indices.
> > > Much more efficient.
> > >
> > > So the year index won't have any removal and you will end up to just
> > > remove weekly indices which is super fast and doesn't generate a lot of IO
> > > (but ttl does)
> > >
> > > HTH
> > >
> > > David
> > >
> > > Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > >
> > > My data contains a weekly ttl so my ES index will loose older data,
> > > however I want to have another accumulative large index containing a year
> > > of my data... How can I instruct ES/sync my second index without affecting
> > > ttls in the first index? ( I don't want deleted docs from first index to be
> > > deleted in the second)
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to a topic in the
> > > Google Groups "elasticsearch" group.
> > > To unsubscribe from this topic, visit
> > > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > To unsubscribe from this group and all its topics, send an email to
> > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr)
> > > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer)
> > > .
> > >
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > --Behrad
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com)
> > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit
> > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > To unsubscribe from this group and all its topics, send an email to
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr)
> > [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer)
> > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> --Behrad
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr)
> [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
--Behrad
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau\_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 14, 2015, 5:32pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/8 "2015-01-14T17:32:16Z")
I see. Just wondering why you are sending things to couchbase then. Why not using Redis or RabbitMQ or any message queue System. I mean that it sounds like you are using couchbase as a queue for elasticsearch.
Or if you want a Backup in couchbase, could you imagine sending your data from your application to couchbase and elasticsearch at the same time?
I think you are trying to twist here the couchbase plugin which is designed to keep the database in sync with elasticsearch which is obviously not what you are looking for.
My 2 cents.
David
> Le 14 janv. 2015 à 18:18, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> 2015-01-14 18:25 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
>
> > Definitely scan and scroll API is for that.
> > Some clients provide that feature out of the box.
> >
> > That said, reading from couchbase is something super easy. So I guess you could may be replicate your data from couchbase twice using 2 XDCR configs one for the weekly index and the other one for the year index. Is that something doable?
>
> Unfortunately not David, XDCR blinding syncs the source db with remotes (any deleted data will also be deleted from the remote) and we can't let the source db to grow (about 1.3 billion records/month) so we are using couchbase TTLs to store a week or so,
> however we'd like to have online searching/access to a multiple month (i hope by utilizing ES) and I'm seeking the best Enterprise approach to this.
>
> However I haven't any clue for capacity planning of ES for 5, 6 billions of records.
>
> > --
> > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
> > @dadoonet | @elasticsearchfr | @scrutmydocs
> >
> > > Le 14 janv. 2015 à 14:51, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > >
> > > I'll recheck elasticsearch-transport-couchbase again David, another two related questions comes to my mind to ask,
> > >
> > > 1. Is there any best practice to replicate/sync/duplicate an existing index to a new index by ES?
> > > 2. Considering I have an existing index with data persist enabled on ES, Can I use that to create a new index with a customer defined mapping on my data on ES without the need of my primary database? (does sroll api help with this?)
> > >
> > > 2015-01-14 17:00 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > >
> > > > May be you could use different mapping for the year index and the week index?
> > > > One with ttl disabled and the other one with ttl enabled?
> > > >
> > > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
> > > >
> > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
> > > > @dadoonet | @elasticsearchfr | @scrutmydocs
> > > >
> > > > > Le 14 janv. 2015 à 14:21, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > > >
> > > > > I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
> > > > >
> > > > > 2015-01-14 16:43 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > > > >
> > > > > > Don't use ttl for that kind of usage but prefer time based indices.
> > > > > > Much more efficient.
> > > > > >
> > > > > > So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)
> > > > > >
> > > > > > HTH
> > > > > >
> > > > > > David
> > > > > >
> > > > > > > Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > > > > >
> > > > > > > My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com).
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > > > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr).
> > > > > >
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > >
> > > > > --
> > > > > --Behrad
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com).
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr).
> > > >
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > --Behrad
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com).
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr).
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> ## -- --Behrad
>
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau\_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Behrad\_Zari](https://discuss.elastic.co/u/Behrad_Zari)
#### Post date: [January 14, 2015, 6:38pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/9 "2015-01-14T18:38:17Z")
This is our persistent storage, but the working set is a multiple days,
older data only is needed for search and audits, the whole data is being
backed up and archived by a separate process, so we don't need extra large
hard disks (each billion records gets to 600GB) Redis is not designed for
persistency and Message Queues are slow when persistency is envolved, if we
were using a Message Queue, we also should have persist our data on a
durable external storage. I want to have a small working set of data of a
week, and a workhouse of multi months of past data. (I can't see why to
keep multi billions of passive data in Couchbase which should be searched
by ES!?)
2015-01-14 21:02 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> I see. Just wondering why you are sending things to couchbase then. Why
> not using Redis or RabbitMQ or any message queue System. I mean that it
> sounds like you are using couchbase as a queue for elasticsearch.
>
> Or if you want a Backup in couchbase, could you imagine sending your data
> from your application to couchbase and elasticsearch at the same time?
>
> I think you are trying to twist here the couchbase plugin which is
> designed to keep the database in sync with elasticsearch which is obviously
> not what you are looking for.
>
> My 2 cents.
>
> David
>
> Le 14 janv. 2015 à 18:18, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> 2015-01-14 18:25 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
>
> > Definitely scan and scroll API is for that.
> > Some clients provide that feature out of the box.
> >
> > That said, reading from couchbase is something super easy. So I guess you
> > could may be replicate your data from couchbase twice using 2 XDCR configs
> > one for the weekly index and the other one for the year index. Is that
> > something doable?
>
> Unfortunately not David, XDCR blinding syncs the source db with remotes
> (any deleted data will also be deleted from the remote) and we can't let
> the source db to grow (about 1.3 billion records
> /month) so we are using couchbase TTLs to store a week or so,
> however we'd like to have online searching/access to a multiple month (i
> hope by utilizing ES) and I'm seeking the best Enterprise approach to this.
>
> However I haven't any clue for capacity planning of ES for 5, 6 billions
> of records.
>
> > --
> > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)
> > [http://Elasticsearch.com](http://Elasticsearch.com)_
> > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr
> > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs
> > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> >
> > Le 14 janv. 2015 à 14:51, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> >
> > I'll recheck elasticsearch-transport-couchbase again David, another two
> > related questions comes to my mind to ask,
> >
> > 1. Is there any best practice to replicate/sync/duplicate an existing
> > index to a new index by ES?
> > 2. Considering I have an existing index with data persist enabled on ES,
> > Can I use that to create a new index with a customer defined mapping on my
> > data on ES without the need of my primary database? (does sroll api help
> > with this?)
> >
> > 2015-01-14 17:00 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> >
> > > May be you could use different mapping for the year index and the week
> > > index?
> > > One with ttl disabled and the other one with ttl enabled?
> > >
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
> > >
> > > --
> > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)
> > > [http://elasticsearch.com/](http://elasticsearch.com/)_
> > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr
> > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs
> > > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> > >
> > > Le 14 janv. 2015 à 14:21, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > >
> > > I am actually using Couchbase ES plugin David, this is my database
> > > requirement that data has a weekly ttl, so when data is removed in
> > > Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
> > >
> > > 2015-01-14 16:43 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > >
> > > > Don't use ttl for that kind of usage but prefer time based indices.
> > > > Much more efficient.
> > > >
> > > > So the year index won't have any removal and you will end up to just
> > > > remove weekly indices which is super fast and doesn't generate a lot of IO
> > > > (but ttl does)
> > > >
> > > > HTH
> > > >
> > > > David
> > > >
> > > > Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > >
> > > > My data contains a weekly ttl so my ES index will loose older data,
> > > > however I want to have another accumulative large index containing a year
> > > > of my data... How can I instruct ES/sync my second index without affecting
> > > > ttls in the first index? ( I don't want deleted docs from first index to be
> > > > deleted in the second)
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > You received this message because you are subscribed to a topic in the
> > > > Google Groups "elasticsearch" group.
> > > > To unsubscribe from this topic, visit
> > > > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe)
> > > > .
> > > > To unsubscribe from this group and all its topics, send an email to
> > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr)
> > > > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer)
> > > > .
> > > >
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > --Behrad
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to a topic in the
> > > Google Groups "elasticsearch" group.
> > > To unsubscribe from this topic, visit
> > > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > To unsubscribe from this group and all its topics, send an email to
> > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr)
> > > [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer)
> > > .
> > >
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > --Behrad
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com)
> > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit
> > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > To unsubscribe from this group and all its topics, send an email to
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr)
> > [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer)
> > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> --Behrad
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau\_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau\_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr)
> [https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
--Behrad
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N\_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 14, 2015, 6:53pm UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/10 "2015-01-14T18:53:55Z")
Ok. I think that you need to use something else than the transport plugin for your specific use case which is not a database sync feature.
Any chance you could push from your source to couchbase AND elasticsearch at the same time? Or write a batch which reads couchbase and write to elasticsearch.
I don't think there is yet an input for logstash but I might be wrong...
--
David 😉
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> Le 14 janv. 2015 à 19:38, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> This is our persistent storage, but the working set is a multiple days, older data only is needed for search and audits, the whole data is being backed up and archived by a separate process, so we don't need extra large hard disks (each billion records gets to 600GB) Redis is not designed for persistency and Message Queues are slow when persistency is envolved, if we were using a Message Queue, we also should have persist our data on a durable external storage. I want to have a small working set of data of a week, and a workhouse of multi months of past data. (I can't see why to keep multi billions of passive data in Couchbase which should be searched by ES!?)
>
> 2015-01-14 21:02 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
>
> > I see. Just wondering why you are sending things to couchbase then. Why not using Redis or RabbitMQ or any message queue System. I mean that it sounds like you are using couchbase as a queue for elasticsearch.
> >
> > Or if you want a Backup in couchbase, could you imagine sending your data from your application to couchbase and elasticsearch at the same time?
> >
> > I think you are trying to twist here the couchbase plugin which is designed to keep the database in sync with elasticsearch which is obviously not what you are looking for.
> >
> > My 2 cents.
> >
> > David
> >
> > > Le 14 janv. 2015 à 18:18, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > >
> > > 2015-01-14 18:25 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > >
> > > > Definitely scan and scroll API is for that.
> > > > Some clients provide that feature out of the box.
> > > >
> > > > That said, reading from couchbase is something super easy. So I guess you could may be replicate your data from couchbase twice using 2 XDCR configs one for the weekly index and the other one for the year index. Is that something doable?
> > >
> > > Unfortunately not David, XDCR blinding syncs the source db with remotes (any deleted data will also be deleted from the remote) and we can't let the source db to grow (about 1.3 billion records/month) so we are using couchbase TTLs to store a week or so,
> > > however we'd like to have online searching/access to a multiple month (i hope by utilizing ES) and I'm seeking the best Enterprise approach to this.
> > >
> > > However I haven't any clue for capacity planning of ES for 5, 6 billions of records.
> > >
> > > > --
> > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
> > > > @dadoonet | @elasticsearchfr | @scrutmydocs
> > > >
> > > > > Le 14 janv. 2015 à 14:51, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > > >
> > > > > I'll recheck elasticsearch-transport-couchbase again David, another two related questions comes to my mind to ask,
> > > > >
> > > > > 1. Is there any best practice to replicate/sync/duplicate an existing index to a new index by ES?
> > > > > 2. Considering I have an existing index with data persist enabled on ES, Can I use that to create a new index with a customer defined mapping on my data on ES without the need of my primary database? (does sroll api help with this?)
> > > > >
> > > > > 2015-01-14 17:00 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > > > >
> > > > > > May be you could use different mapping for the year index and the week index?
> > > > > > One with ttl disabled and the other one with ttl enabled?
> > > > > >
> > > > > > ## [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
> > > > > >
> > > > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)
> > > > > > @dadoonet | @elasticsearchfr | @scrutmydocs
> > > > > >
> > > > > > > Le 14 janv. 2015 à 14:21, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > > > > >
> > > > > > > I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
> > > > > > >
> > > > > > > 2015-01-14 16:43 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > > > > > >
> > > > > > > > Don't use ttl for that kind of usage but prefer time based indices.
> > > > > > > > Much more efficient.
> > > > > > > >
> > > > > > > > So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)
> > > > > > > >
> > > > > > > > HTH
> > > > > > > >
> > > > > > > > David
> > > > > > > >
> > > > > > > > > Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > > > > > > >
> > > > > > > > > My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com).
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > >
> > > > > > > > --
> > > > > > > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > > > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > > > > > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr).
> > > > > > > >
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > >
> > > > > > > --
> > > > > > > --Behrad
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com).
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > > > > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > > > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr).
> > > > > >
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > >
> > > > > --
> > > > > --Behrad
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com).
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr).
> > > >
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > ## -- --Behrad
> > >
> > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau\_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com).
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr).
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> ## -- --Behrad
>
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N\_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com).
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Behrad\_Zari](https://discuss.elastic.co/u/Behrad_Zari)
#### Post date: [January 15, 2015, 6:43am UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/11 "2015-01-15T06:43:52Z")
2015-01-14 22:23 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> Ok. I think that you need to use something else than the transport plugin
> for your specific use case which is not a database sync feature.
>
> Any chance you could push from your source to couchbase AND elasticsearch
> at the same time? Or write a batch which reads couchbase and write to
> elasticsearch.
I can stream from Couchbase views into ES by a hand made script., but I
was elaborating available toolset to achieve this automatically.
> I don't think there is yet an input for logstash but I might be wrong...
>
> --
> David 😉
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 14 janv. 2015 à 19:38, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
>
> This is our persistent storage, but the working set is a multiple days,
> older data only is needed for search and audits, the whole data is being
> backed up and archived by a separate process, so we don't need extra large
> hard disks (each billion records gets to 600GB) Redis is not designed for
> persistency and Message Queues are slow when persistency is envolved, if we
> were using a Message Queue, we also should have persist our data on a
> durable external storage. I want to have a small working set of data of a
> week, and a workhouse of multi months of past data. (I can't see why to
> keep multi billions of passive data in Couchbase which should be searched
> by ES!?)
>
> 2015-01-14 21:02 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
>
> > I see. Just wondering why you are sending things to couchbase then. Why
> > not using Redis or RabbitMQ or any message queue System. I mean that it
> > sounds like you are using couchbase as a queue for elasticsearch.
> >
> > Or if you want a Backup in couchbase, could you imagine sending your data
> > from your application to couchbase and elasticsearch at the same time?
> >
> > I think you are trying to twist here the couchbase plugin which is
> > designed to keep the database in sync with elasticsearch which is obviously
> > not what you are looking for.
> >
> > My 2 cents.
> >
> > David
> >
> > Le 14 janv. 2015 à 18:18, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> >
> > 2015-01-14 18:25 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> >
> > > Definitely scan and scroll API is for that.
> > > Some clients provide that feature out of the box.
> > >
> > > That said, reading from couchbase is something super easy. So I guess
> > > you could may be replicate your data from couchbase twice using 2 XDCR
> > > configs one for the weekly index and the other one for the year index. Is
> > > that something doable?
> >
> > Unfortunately not David, XDCR blinding syncs the source db with remotes
> > (any deleted data will also be deleted from the remote) and we can't let
> > the source db to grow (about 1.3 billion records
> > /month) so we are using couchbase TTLs to store a week or so,
> > however we'd like to have online searching/access to a multiple month (i
> > hope by utilizing ES) and I'm seeking the best Enterprise approach to this.
> >
> > However I haven't any clue for capacity planning of ES for 5, 6 billions
> > of records.
> >
> > > --
> > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)
> > > [http://Elasticsearch.com](http://Elasticsearch.com)_
> > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr
> > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs
> > > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> > >
> > > Le 14 janv. 2015 à 14:51, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > >
> > > I'll recheck elasticsearch-transport-couchbase again David, another two
> > > related questions comes to my mind to ask,
> > >
> > > 1. Is there any best practice to replicate/sync/duplicate an existing
> > > index to a new index by ES?
> > > 2. Considering I have an existing index with data persist enabled on ES,
> > > Can I use that to create a new index with a customer defined mapping on my
> > > data on ES without the need of my primary database? (does sroll api help
> > > with this?)
> > >
> > > 2015-01-14 17:00 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > >
> > > > May be you could use different mapping for the year index and the week
> > > > index?
> > > > One with ttl disabled and the other one with ttl enabled?
> > > >
> > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)
> > > >
> > > > --
> > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)
> > > > [http://elasticsearch.com/](http://elasticsearch.com/)_
> > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr
> > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs
> > > > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)
> > > >
> > > > Le 14 janv. 2015 à 14:21, Behrad [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > >
> > > > I am actually using Couchbase ES plugin David, this is my database
> > > > requirement that data has a weekly ttl, so when data is removed in
> > > > Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
> > > >
> > > > 2015-01-14 16:43 GMT+03:30 David Pilato [david@pilato.fr](mailto:david@pilato.fr):
> > > >
> > > > > Don't use ttl for that kind of usage but prefer time based indices.
> > > > > Much more efficient.
> > > > >
> > > > > So the year index won't have any removal and you will end up to just
> > > > > remove weekly indices which is super fast and doesn't generate a lot of IO
> > > > > (but ttl does)
> > > > >
> > > > > HTH
> > > > >
> > > > > David
> > > > >
> > > > > Le 14 janv. 2015 à 14:01, Behrad Zari [behradz@gmail.com](mailto:behradz@gmail.com) a écrit :
> > > > >
> > > > > My data contains a weekly ttl so my ES index will loose older data,
> > > > > however I want to have another accumulative large index containing a year
> > > > > of my data... How can I instruct ES/sync my second index without affecting
> > > > > ttls in the first index? ( I don't want deleted docs from first index to be
> > > > > deleted in the second)
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > To view this discussion on the web visit
> > > > > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com)
> > > > > [https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > .
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to a topic in the
> > > > > Google Groups "elasticsearch" group.
> > > > > To unsubscribe from this topic, visit
> > > > > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe)
> > > > > .
> > > > > To unsubscribe from this group and all its topics, send an email to
> > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > > To view this discussion on the web visit
> > > > > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr)
> > > > > [https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer)
> > > > > .
> > > > >
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > --Behrad
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH\_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > You received this message because you are subscribed to a topic in the
> > > > Google Groups "elasticsearch" group.
> > > > To unsubscribe from this topic, visit
> > > > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe)
> > > > .
> > > > To unsubscribe from this group and all its topics, send an email to
> > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > > To view this discussion on the web visit
> > > > [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr)
> > > > [https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer)
> > > > .
> > > >
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > --Behrad
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg\_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to a topic in the
> > > Google Groups "elasticsearch" group.
> > > To unsubscribe from this topic, visit
> > > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > > To unsubscribe from this group and all its topics, send an email to
> > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr)
> > > [https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer)
> > > .
> > >
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > --Behrad
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau\_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com)
> > [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau\_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit
> > [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> > To unsubscribe from this group and all its topics, send an email to
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr)
> > [https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm_medium=email&utm_source=footer)
> > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> --Behrad
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N\_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N\_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr)
> [https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
--Behrad
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFPQ35HeT9GotRPido6aZ%3D8665Ki%3D\_uP%3DxT33gbkHovOA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFPQ35HeT9GotRPido6aZ%3D8665Ki%3D_uP%3DxT33gbkHovOA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:38am UTC](https://discuss.elastic.co/t/accumulate-a-ttl-index-into-a-large-backup-index/21635/12 "2017-07-06T00:38:47Z")