This is our persistent storage, but the working set is a multiple days,
older data only is needed for search and audits, the whole data is being
backed up and archived by a separate process, so we don't need extra large
hard disks (each billion records gets to 600GB) Redis is not designed for
persistency and Message Queues are slow when persistency is envolved, if we
were using a Message Queue, we also should have persist our data on a
durable external storage. I want to have a small working set of data of a
week, and a workhouse of multi months of past data. (I can't see why to
keep multi billions of passive data in Couchbase which should be searched
by ES!?)
2015-01-14 21:02 GMT+03:30 David Pilato david@pilato.fr:
I see. Just wondering why you are sending things to couchbase then. Why
not using Redis or RabbitMQ or any message queue System. I mean that it
sounds like you are using couchbase as a queue for elasticsearch.
Or if you want a Backup in couchbase, could you imagine sending your data
from your application to couchbase and elasticsearch at the same time?
I think you are trying to twist here the couchbase plugin which is
designed to keep the database in sync with elasticsearch which is obviously
not what you are looking for.
My 2 cents.
David
Le 14 janv. 2015 à 18:18, Behrad behradz@gmail.com a écrit :
2015-01-14 18:25 GMT+03:30 David Pilato david@pilato.fr:
Definitely scan and scroll API is for that.
Some clients provide that feature out of the box.
That said, reading from couchbase is something super easy. So I guess you
could may be replicate your data from couchbase twice using 2 XDCR configs
one for the weekly index and the other one for the year index. Is that
something doable?
Unfortunately not David, XDCR blinding syncs the source db with remotes
(any deleted data will also be deleted from the remote) and we can't let
the source db to grow (about 1.3 billion records
/month) so we are using couchbase TTLs to store a week or so,
however we'd like to have online searching/access to a multiple month (i
hope by utilizing ES) and I'm seeking the best Enterprise approach to this.
However I haven't any clue for capacity planning of ES for 5, 6 billions
of records.
--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs
Le 14 janv. 2015 à 14:51, Behrad behradz@gmail.com a écrit :
I'll recheck elasticsearch-transport-couchbase again David, another two
related questions comes to my mind to ask,
- Is there any best practice to replicate/sync/duplicate an existing
index to a new index by ES?
- Considering I have an existing index with data persist enabled on ES,
Can I use that to create a new index with a customer defined mapping on my
data on ES without the need of my primary database? (does sroll api help
with this?)
2015-01-14 17:00 GMT+03:30 David Pilato david@pilato.fr:
May be you could use different mapping for the year index and the week
index?
One with ttl disabled and the other one with ttl enabled?
Elasticsearch Platform — Find real-time answers at scale | Elastic
--
David Pilato | Technical Advocate | Elasticsearch.com
http://elasticsearch.com/
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs
Le 14 janv. 2015 à 14:21, Behrad behradz@gmail.com a écrit :
I am actually using Couchbase ES plugin David, this is my database
requirement that data has a weekly ttl, so when data is removed in
Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES.
2015-01-14 16:43 GMT+03:30 David Pilato david@pilato.fr:
Don't use ttl for that kind of usage but prefer time based indices.
Much more efficient.
So the year index won't have any removal and you will end up to just
remove weekly indices which is super fast and doesn't generate a lot of IO
(but ttl does)
HTH
David
Le 14 janv. 2015 à 14:01, Behrad Zari behradz@gmail.com a écrit :
My data contains a weekly ttl so my ES index will loose older data,
however I want to have another accumulative large index containing a year
of my data... How can I instruct ES/sync my second index without affecting
ttls in the first index? ( I don't want deleted docs from first index to be
deleted in the second)
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
--Behrad
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
--Behrad
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
--Behrad
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
--Behrad
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
