Like in the answer from @tylerjl in my previous post (link) we'd like to arrange the costumer with an filtered aliases.
Does someone has experience with a filtered alias against so many and big datas and went into problems ?
To avoid to create unnecessary shards we would like make some kind of requests on the current index with only one or a few shards and if a limit is reached, create a new one. (for example put all in the index "allcostumers_2015_05-1" and if the limit is reached create an index "allcostumers_2015_05-2" , 2015 for the year and 05 for the month)
The Questions are how to set a limit?
By Size and the number documents ?
What would be a good limit e.g. 100 Million documents or 50 GB for a shard?
Does somebody created something like this and would share their experience ?
Some corresponding informations to our documents and system:
We will have about 216 millionen documents with lets say about 1,6 TB for this year.
We have about 1000 Costumer and we'd like to use an index per month.
3 dedicated master. 3 clients for which import and 2 clients for searches .
9 Data nodes with 8 cores, 8-16 Gb Ram and 1 TB each.