I have an elk siem (single node) version 8.7.1 in production.
On it is a 5TB data partition with about 90% disk usage.
I would like to allow lifecycle policies to rotate data every 180 days (6 months). It's possible? What would be the best solution for this? Can you help me?
You could create an Index Lifecycle Policy to do that, since you have only one node you would have only the hot phase and the delete phase, then you could configure your policy to delete your indices after the number of days you want.
But this depends on how are you creating your indices, are you using daily indices?
For example, the following policy would delete your indices after they reach 180 days.
PUT _ilm/policy/policyName
{
"policy": {
"phases": {
"hot": {
"actions": {},
"min_age": "0ms"
},
"delete": {
"min_age": "180d",
"actions": {
"delete": {}
}
}
}
}
}
@Thales_Eduardo What are you using to collect the logs?
Do you have a sense of how many GB / Day you are collecting?
You can edit that policy and add the Delete Phase?
It will warn you you are editing a "Managed Policy" which is fine...
And of course, I do have to say with a single node you are at risk of data loss if something were to happen to that Disk, Host, VM, Node etc...
What are you using to collect the logs?
R: I'm using this Docker environment setup which uses Logstash. (GitHub - sherifabdlnaby/elastdocker: 🐳 Elastic Stack (ELK) v8+ on Docker with Compose. Pre-configured out of the box to enable Logging, Metrics, APM, Alerting, ML, and SIEM features. Up with a Single Command.)
Do you have a sense of how many GB / Day you are collecting?
R: On average 30Gb
You can edit that policy and add the Delete Phase? It will warn you you are editing a "Managed Policy" which is fine...
R: OK. So I'm going to edit it and add the Delete phase.
And of course, I do have to say with a single node you are at risk of data loss if something were to happen to that Disk, Host, VM, Node etc...
R: OK. I know.
1 Like
That's not really a great idea, I would go with monthly rotation and then delete after 6 months.
Cause what happens if your incoming data volumes increase and a single 6 month index becomes 10X the size?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.