Thanks for your feedback Shay.
There can be few high traffic tenants where I would need to store as much as 20 million entries in a day.
As routing works on a single shard, will it be fine to store so much data on a single shard. I am still in investigation mode and do not have a cluster to verify the performance with respect to data volume and shard allocations.
If it is standard to store this much data on a single shard, I would move ahead with this assumption.
On Jun 25, 2012, at 5:14 AM, Shay Banon wrote:
Note, is considerably cheaper to delete an index than delete data from an index.
Having 3000 shards is possible, but will "cost" you in terms of the number of nodes you will need to start in order to support it. What I would do is do index per day, and on that index, have several shards holding data for all tenants, and using routing (the tenant "name") to direct specific tenant data to a shard.
Here is the presentation I gave that explains it: https://speakerdeck.com/u/kimchy/p/elasticsearch-big-data-search-analytics.
On Sun, Jun 24, 2012 at 9:51 AM, Alexander Reelsen alr@spinscale.de wrote:
Hi Ashish,On Fri, Jun 22, 2012 at 11:20 PM, Ashish Nigam ashish@skyhighnetworks.com wrote:
To remove docs on nightly basis from the index, is there any good way to perform soft delete, i.e move a day worth of data from an index to another index?You might want to check the TTL feature for this, so your docs get automatically deleted after 30 days.
Elasticsearch Platform — Find real-time answers at scale | Elastic
--Alexander