Overview
i have elasticsearch 8.11 with no replicas right now, only 5 hot nodes and 2 frozen that's it.
Here is index lifecycle policy is this
PUT _ilm/policy/ConrainerLogs
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "3h",
"max_primary_shard_size": "50gb"
},
"set_priority": {
"priority": 100
}
}
},
"frozen": {
"min_age": "6h",
"actions": {
"searchable_snapshot": {
"snapshot_repository": "s3_repository",
"force_merge_index": true
}
}
},
"delete": {
"min_age": "10h",
"actions": {
"delete": {
"delete_searchable_snapshot": true
},
"wait_for_snapshot": {
"policy": "1-hour-snapshots"
}
}
}
}
}
}
i have 2 Snapshot Lifecyle policies one is daily which runs at schduled at 1:30 every day means at particualr time which exlucdes this datastream "contianer-logs" and includes all other. so i have created another one for this particualr one that i want to manage it is given below for take backup
the issue i am facing is my index in hot nodes are not being rollover to new indexes when the size is reached. i have 2-5 TB ingestion per day in that particualr index , so idea is to have these index as searchable and then after 10 hour delete the indexes after checking if the snapshot exits if not then take the snapshot and delete the index.
but issue is when it check it say can't rollover the datastream is being snapshoted due to that the current index new one could not rollover to new index in hot phase and size increases to TB's how to handle this expertly so my index never goes above 50 GB and snapshot each index on the time and never miss any data.