Rollover of Index and delete delta data as per customer configuration

Elastic Stack Elasticsearch
1

Hi Experts,

I have customer requirement where each customer can opt XGB of data or Y days of data and n no of records. Customer is saying they only want this much, if more data is coming, please delete oldest among all. How can we achieve using ILM? Here we do not want so much of rollover as the index and shard count increases, and we can trap in over sharding issues. Here we have separate index for each customer. Currently we are using 7.16 ES.

2

ILM works by deleting complete indices, so in order to use this you would need to create time-based indices, e.g. using rollover. To answer this there are a few additional questions that would be useful to have answers to:

How many customers do you need to support?

How much data does each customer typically have?

How long retention period do you need to support?

Do you have control over the data being ingested and the mappings?

Are you giving access to this data through an API layer that you control or can the customer interact directly with the data, e.g. through Kibana?

3

Please find the ans inline

image001.png

image002.png

image003.png

image006.png

image007.png

image008.png

image009.jpg
image009.jpg728×90 38.8 KB

image010.png

4

ILM works by deleting complete indices, so in order to use this you would need to create time-based indices, e.g. using rollover. To answer this there are a few additional questions that would be useful to have answers to:

How many customers do you need to support?

Around 1000 customers

How much data does each customer typically have?

200 customers can have 40000 per day , 400 customers can have 4 million records per day, 400 customers can have 40 million records per day

Size of 1 record is 20 KB/record

How long retention period do you need to support?

30 days, 1 month, 3 months , 9 months and 12 months

Do you have control over the data being ingested and the mappings?

Yes

Are you giving access to this data through an API layer that you control or can the customer interact directly with the data, e.g. through Kibana?

We have UI and customer will use this ui to interact with data.

Please note we are using es 7.16

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.