That'll hit limits eventually, each shard can only hold ~2^32 docs.
What if I were to include a TTL on the document ?
Just go with time based indices and tag them with the various IDs
I`m not sure what you mean by this. Like, create a new monthly index or something like that ? This seems strange to me.
That said, I came up with a minimal example yesterday. I have a single index and tagged each document with a publisherId. I could request documents with associated publishers using a Terms query pretty easily.
I had 400k documents in my single index and had no special shard logic at first. On a second iteration I split my documents into two shards. When I did this I saw no performance increases and was wondering if this extra step was worth the risk. Perhaps I was doing it wrong ?
Bulk insert Logic
for (int p = 0; p < publisherCount; p++)
for (int r = 0; r < recordCount; r++)
var ops = new BulkCreateDescriptor<NewsModel>();
ops.Routing(route); // "1" or "2"
id = Guid.NewGuid().ToString(),
created = DateTime.UtcNow.Subtract(TimeSpan.FromHours(rand.Next(1, publisherCount * recordCount))),
publisherId = p + publisherStart
var response = await client.BulkAsync(descriptor);
List<int> friends = new List<int>();
for (int i = 0; i < friendCount; i++)
var search = new SearchDescriptor<NewsModel>();
search.Sort(so => so.Descending(a => a.created));
search.Query(q => q.Terms(t => t.Field(f => f.publisherId).Terms<int>(friends)));
var result = client.Search<NewsModel>(search);