Accumulate a ttl index into a large backup index

My data contains a weekly ttl so my ES index will loose older data, however
I want to have another accumulative large index containing a year of my
data... How can I instruct ES/sync my second index without affecting ttls
in the first index? ( I don't want deleted docs from first index to be
deleted in the second)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

I am actually using Couchbase ES plugin David, this is my database
requirement that data has a weekly ttl, so when data is removed in
Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just
remove weekly indices which is super fast and doesn't generate a lot of IO
(but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data,
however I want to have another accumulative large index containing a year
of my data... How can I instruct ES/sync my second index without affecting
ttls in the first index? ( I don't want deleted docs from first index to be
deleted in the second)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

May be you could use different mapping for the year index and the week index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :

I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato <david@pilato.fr mailto:david@pilato.fr>:
Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari <behradz@gmail.com mailto:behradz@gmail.com> a écrit :

My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

I'll recheck elasticsearch-transport-couchbase again David, another two
related questions comes to my mind to ask,

  1. Is there any best practice to replicate/sync/duplicate an existing index
    to a new index by ES?
  2. Considering I have an existing index with data persist enabled on ES,
    Can I use that to create a new index with a customer defined mapping on my
    data on ES without the need of my primary database? (does sroll api help
    with this?)

2015-01-14 17:00 GMT+03:30 David Pilato david@pilato.fr:

May be you could use different mapping for the year index and the week
index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :

I am actually using Couchbase ES plugin David, this is my database
requirement that data has a weekly ttl, so when data is removed in
Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just
remove weekly indices which is super fast and doesn't generate a lot of IO
(but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data,
however I want to have another accumulative large index containing a year
of my data... How can I instruct ES/sync my second index without affecting
ttls in the first index? ( I don't want deleted docs from first index to be
deleted in the second)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.

That said, reading from couchbase is something super easy. So I guess you could may be replicate your data from couchbase twice using 2 XDCR configs one for the weekly index and the other one for the year index. Is that something doable?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:51, Behrad behradz@gmail.com a écrit :

I'll recheck elasticsearch-transport-couchbase again David, another two related questions comes to my mind to ask,

  1. Is there any best practice to replicate/sync/duplicate an existing index to a new index by ES?
  2. Considering I have an existing index with data persist enabled on ES, Can I use that to create a new index with a customer defined mapping on my data on ES without the need of my primary database? (does sroll api help with this?)

2015-01-14 17:00 GMT+03:30 David Pilato <david@pilato.fr mailto:david@pilato.fr>:
May be you could use different mapping for the year index and the week index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

David Pilato | Technical Advocate | Elasticsearch.com http://elasticsearch.com/
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad <behradz@gmail.com mailto:behradz@gmail.com> a écrit :

I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato <david@pilato.fr mailto:david@pilato.fr>:
Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari <behradz@gmail.com mailto:behradz@gmail.com> a écrit :

My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

2015-01-14 18:25 GMT+03:30 David Pilato david@pilato.fr:

Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.

That said, reading from couchbase is something super easy. So I guess you
could may be replicate your data from couchbase twice using 2 XDCR configs
one for the weekly index and the other one for the year index. Is that
something doable?

Unfortunately not David, XDCR blinding syncs the source db with remotes
(any deleted data will also be deleted from the remote) and we can't let
the source db to grow (about 1.3 billion records​
/month) so we are using couchbase TTLs to store a week or so,
however we'd like to have online searching/access to a multiple month (i
hope by utilizing ES)​ and I'm seeking the best Enterprise approach to this.

However I haven't any clue for capacity planning of ES for 5, 6 billions of
records.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:51, Behrad behradz@gmail.com a écrit :

I'll recheck elasticsearch-transport-couchbase again David, another two
related questions comes to my mind to ask,

  1. Is there any best practice to replicate/sync/duplicate an existing
    index to a new index by ES?
  2. Considering I have an existing index with data persist enabled on ES,
    Can I use that to create a new index with a customer defined mapping on my
    data on ES without the need of my primary database? (does sroll api help
    with this?)

2015-01-14 17:00 GMT+03:30 David Pilato david@pilato.fr:

May be you could use different mapping for the year index and the week
index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

--
David Pilato | Technical Advocate | Elasticsearch.com
http://elasticsearch.com/

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :

I am actually using Couchbase ES plugin David, this is my database
requirement that data has a weekly ttl, so when data is removed in
Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just
remove weekly indices which is super fast and doesn't generate a lot of IO
(but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data,
however I want to have another accumulative large index containing a year
of my data... How can I instruct ES/sync my second index without affecting
ttls in the first index? ( I don't want deleted docs from first index to be
deleted in the second)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I see. Just wondering why you are sending things to couchbase then. Why not using Redis or RabbitMQ or any message queue System. I mean that it sounds like you are using couchbase as a queue for elasticsearch.

Or if you want a Backup in couchbase, could you imagine sending your data from your application to couchbase and elasticsearch at the same time?

I think you are trying to twist here the couchbase plugin which is designed to keep the database in sync with elasticsearch which is obviously not what you are looking for.

My 2 cents.

David

Le 14 janv. 2015 à 18:18, Behrad behradz@gmail.com a écrit :

2015-01-14 18:25 GMT+03:30 David Pilato david@pilato.fr:

Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.

That said, reading from couchbase is something super easy. So I guess you could may be replicate your data from couchbase twice using 2 XDCR configs one for the weekly index and the other one for the year index. Is that something doable?

Unfortunately not David, XDCR blinding syncs the source db with remotes (any deleted data will also be deleted from the remote) and we can't let the source db to grow (about 1.3 billion records​/month) so we are using couchbase TTLs to store a week or so,
however we'd like to have online searching/access to a multiple month (i hope by utilizing ES)​ and I'm seeking the best Enterprise approach to this.

However I haven't any clue for capacity planning of ES for 5, 6 billions of records.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 14 janv. 2015 à 14:51, Behrad behradz@gmail.com a écrit :

I'll recheck elasticsearch-transport-couchbase again David, another two related questions comes to my mind to ask,

  1. Is there any best practice to replicate/sync/duplicate an existing index to a new index by ES?
  2. Considering I have an existing index with data persist enabled on ES, Can I use that to create a new index with a customer defined mapping on my data on ES without the need of my primary database? (does sroll api help with this?)

2015-01-14 17:00 GMT+03:30 David Pilato david@pilato.fr:

May be you could use different mapping for the year index and the week index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :

I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

This is our persistent storage, but the working set is a multiple days,
older data only is needed for search and audits, the whole data is being
backed up and archived by a separate process, so we don't need extra large
hard disks (each billion records gets to 600GB) Redis is not designed for
persistency and Message Queues are slow when persistency is envolved, if we
were using a Message Queue, we also should have persist our data on a
durable external storage. I want to have a small working set of data of a
week, and a workhouse of multi months of past data. (I can't see why to
keep multi billions of passive data in Couchbase which should be searched
by ES!?)

2015-01-14 21:02 GMT+03:30 David Pilato david@pilato.fr:

I see. Just wondering why you are sending things to couchbase then. Why
not using Redis or RabbitMQ or any message queue System. I mean that it
sounds like you are using couchbase as a queue for elasticsearch.

Or if you want a Backup in couchbase, could you imagine sending your data
from your application to couchbase and elasticsearch at the same time?

I think you are trying to twist here the couchbase plugin which is
designed to keep the database in sync with elasticsearch which is obviously
not what you are looking for.

My 2 cents.

David

Le 14 janv. 2015 à 18:18, Behrad behradz@gmail.com a écrit :

2015-01-14 18:25 GMT+03:30 David Pilato david@pilato.fr:

Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.

That said, reading from couchbase is something super easy. So I guess you
could may be replicate your data from couchbase twice using 2 XDCR configs
one for the weekly index and the other one for the year index. Is that
something doable?

Unfortunately not David, XDCR blinding syncs the source db with remotes
(any deleted data will also be deleted from the remote) and we can't let
the source db to grow (about 1.3 billion records​
/month) so we are using couchbase TTLs to store a week or so,
however we'd like to have online searching/access to a multiple month (i
hope by utilizing ES)​ and I'm seeking the best Enterprise approach to this.

However I haven't any clue for capacity planning of ES for 5, 6 billions
of records.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:51, Behrad behradz@gmail.com a écrit :

I'll recheck elasticsearch-transport-couchbase again David, another two
related questions comes to my mind to ask,

  1. Is there any best practice to replicate/sync/duplicate an existing
    index to a new index by ES?
  2. Considering I have an existing index with data persist enabled on ES,
    Can I use that to create a new index with a customer defined mapping on my
    data on ES without the need of my primary database? (does sroll api help
    with this?)

2015-01-14 17:00 GMT+03:30 David Pilato david@pilato.fr:

May be you could use different mapping for the year index and the week
index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

--
David Pilato | Technical Advocate | Elasticsearch.com
http://elasticsearch.com/

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :

I am actually using Couchbase ES plugin David, this is my database
requirement that data has a weekly ttl, so when data is removed in
Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just
remove weekly indices which is super fast and doesn't generate a lot of IO
(but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data,
however I want to have another accumulative large index containing a year
of my data... How can I instruct ES/sync my second index without affecting
ttls in the first index? ( I don't want deleted docs from first index to be
deleted in the second)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ok. I think that you need to use something else than the transport plugin for your specific use case which is not a database sync feature.

Any chance you could push from your source to couchbase AND elasticsearch at the same time? Or write a batch which reads couchbase and write to elasticsearch.
I don't think there is yet an input for logstash but I might be wrong...

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 14 janv. 2015 à 19:38, Behrad behradz@gmail.com a écrit :

This is our persistent storage, but the working set is a multiple days, older data only is needed for search and audits, the whole data is being backed up and archived by a separate process, so we don't need extra large hard disks (each billion records gets to 600GB) Redis is not designed for persistency and Message Queues are slow when persistency is envolved, if we were using a Message Queue, we also should have persist our data on a durable external storage. I want to have a small working set of data of a week, and a workhouse of multi months of past data. (I can't see why to keep multi billions of passive data in Couchbase which should be searched by ES!?)

2015-01-14 21:02 GMT+03:30 David Pilato david@pilato.fr:

I see. Just wondering why you are sending things to couchbase then. Why not using Redis or RabbitMQ or any message queue System. I mean that it sounds like you are using couchbase as a queue for elasticsearch.

Or if you want a Backup in couchbase, could you imagine sending your data from your application to couchbase and elasticsearch at the same time?

I think you are trying to twist here the couchbase plugin which is designed to keep the database in sync with elasticsearch which is obviously not what you are looking for.

My 2 cents.

David

Le 14 janv. 2015 à 18:18, Behrad behradz@gmail.com a écrit :

2015-01-14 18:25 GMT+03:30 David Pilato david@pilato.fr:

Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.

That said, reading from couchbase is something super easy. So I guess you could may be replicate your data from couchbase twice using 2 XDCR configs one for the weekly index and the other one for the year index. Is that something doable?

Unfortunately not David, XDCR blinding syncs the source db with remotes (any deleted data will also be deleted from the remote) and we can't let the source db to grow (about 1.3 billion records​/month) so we are using couchbase TTLs to store a week or so,
however we'd like to have online searching/access to a multiple month (i hope by utilizing ES)​ and I'm seeking the best Enterprise approach to this.

However I haven't any clue for capacity planning of ES for 5, 6 billions of records.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 14 janv. 2015 à 14:51, Behrad behradz@gmail.com a écrit :

I'll recheck elasticsearch-transport-couchbase again David, another two related questions comes to my mind to ask,

  1. Is there any best practice to replicate/sync/duplicate an existing index to a new index by ES?
  2. Considering I have an existing index with data persist enabled on ES, Can I use that to create a new index with a customer defined mapping on my data on ES without the need of my primary database? (does sroll api help with this?)

2015-01-14 17:00 GMT+03:30 David Pilato david@pilato.fr:

May be you could use different mapping for the year index and the week index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :

I am actually using Couchbase ES plugin David, this is my database requirement that data has a weekly ttl, so when data is removed in Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just remove weekly indices which is super fast and doesn't generate a lot of IO (but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data, however I want to have another accumulative large index containing a year of my data... How can I instruct ES/sync my second index without affecting ttls in the first index? ( I don't want deleted docs from first index to be deleted in the second)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

2015-01-14 22:23 GMT+03:30 David Pilato david@pilato.fr:

Ok. I think that you need to use something else than the transport plugin
for your specific use case which is not a database sync feature.

Any chance you could push from your source to couchbase AND elasticsearch
at the same time? Or write a batch which reads couchbase and write to
elasticsearch.

​I can stream from Couchbase views into ES by a hand made script., but I
was elaborating available toolset to achieve this automatically.

I don't think there is yet an input for logstash but I might be wrong...

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 14 janv. 2015 à 19:38, Behrad behradz@gmail.com a écrit :

This is our persistent storage, but the working set is a multiple days,
older data only is needed for search and audits, the whole data is being
backed up and archived by a separate process, so we don't need extra large
hard disks (each billion records gets to 600GB) Redis is not designed for
persistency and Message Queues are slow when persistency is envolved, if we
were using a Message Queue, we also should have persist our data on a
durable external storage. I want to have a small working set of data of a
week, and a workhouse of multi months of past data. (I can't see why to
keep multi billions of passive data in Couchbase which should be searched
by ES!?)

2015-01-14 21:02 GMT+03:30 David Pilato david@pilato.fr:

I see. Just wondering why you are sending things to couchbase then. Why
not using Redis or RabbitMQ or any message queue System. I mean that it
sounds like you are using couchbase as a queue for elasticsearch.

Or if you want a Backup in couchbase, could you imagine sending your data
from your application to couchbase and elasticsearch at the same time?

I think you are trying to twist here the couchbase plugin which is
designed to keep the database in sync with elasticsearch which is obviously
not what you are looking for.

My 2 cents.

David

Le 14 janv. 2015 à 18:18, Behrad behradz@gmail.com a écrit :

2015-01-14 18:25 GMT+03:30 David Pilato david@pilato.fr:

Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.

That said, reading from couchbase is something super easy. So I guess
you could may be replicate your data from couchbase twice using 2 XDCR
configs one for the weekly index and the other one for the year index. Is
that something doable?

Unfortunately not David, XDCR blinding syncs the source db with remotes
(any deleted data will also be deleted from the remote) and we can't let
the source db to grow (about 1.3 billion records​
/month) so we are using couchbase TTLs to store a week or so,
however we'd like to have online searching/access to a multiple month (i
hope by utilizing ES)​ and I'm seeking the best Enterprise approach to this.

However I haven't any clue for capacity planning of ES for 5, 6 billions
of records.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:51, Behrad behradz@gmail.com a écrit :

I'll recheck elasticsearch-transport-couchbase again David, another two
related questions comes to my mind to ask,

  1. Is there any best practice to replicate/sync/duplicate an existing
    index to a new index by ES?
  2. Considering I have an existing index with data persist enabled on ES,
    Can I use that to create a new index with a customer defined mapping on my
    data on ES without the need of my primary database? (does sroll api help
    with this?)

2015-01-14 17:00 GMT+03:30 David Pilato david@pilato.fr:

May be you could use different mapping for the year index and the week
index?
One with ttl disabled and the other one with ttl enabled?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html

--
David Pilato | Technical Advocate | Elasticsearch.com
http://elasticsearch.com/

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :

I am actually using Couchbase ES plugin David, this is my database
requirement that data has a weekly ttl, so when data is removed in
Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.

2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:

Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.

So the year index won't have any removal and you will end up to just
remove weekly indices which is super fast and doesn't generate a lot of IO
(but ttl does)

HTH

David

Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :

My data contains a weekly ttl so my ES index will loose older data,
however I want to have another accumulative large index containing a year
of my data... How can I instruct ES/sync my second index without affecting
ttls in the first index? ( I don't want deleted docs from first index to be
deleted in the second)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
--Behrad

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFPQ35HeT9GotRPido6aZ%3D8665Ki%3D_uP%3DxT33gbkHovOA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.