Hello,
i have an elasticsearch cluster with 3 nodes and need help / ideas for the correct index / alias / data structure.
My application have different tenants, each tenant use it's own index to store their transaction data. The transaction data in the indices should be deleted after 3 years.
Sometimes, it can happen, that already inserted documents must be updated.
Some tenants creates 5 million transaction per month, other tenants only 20k transactions per month.
My first idea was to use elasticsearch lifecycle management to create a new index after every 10 million transaction. But sometimes, i need to update transaction and my application don't know, in which index is the transaction stored and it's not possible to store the index in the application (because of high io).
I can create every month a new index per tenant. But after 3 years, i have many small indices / shards, which are a lot overhead for elasticsearch.
Any ideas?
Thanks!