We had disk running full because the "traces-apm.traces-default_policy" did not specify a delete fase.
To make sure the delete fase will be there in the future we defined the traces-apm.traces-default_policy in our stackConfigPolicy.
Now the APM integration cannot upgrade, I believe because of this error we get in Kibana:
illegal_argument_exception Root causes: illegal_argument_exception: Failed to process request [{ "policy" : { "phases" : { "hot" : { "min_age" : "0ms", "actions" : { "set_priority" : { "priority" : 100 }, "rollover" : { "max_age" : "30d", "max_size" : "50gb" } } }, "delete" : { "min_age" : "10d", "actions" : { "delete" : { "delete_searchable_snapshot" : true } } } }, "_meta" : { "package" : { "name" : "apm" }, "managed_by" : "fleet", "managed" : true } } }] **with errors: [[traces-apm.traces-default_policy] set as read-only by [file_settings]]**
In the Console:
PUT _ilm/policy/traces-apm.traces-default_policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"forcemerge": {
"max_num_segments": 1
},
"rollover": {
"max_age": "1d",
"max_size": "50gb"
},
"set_priority": {
"priority": 100
},
"shrink": {
"number_of_shards": 1
}
}
},
"delete": {
"min_age": "14d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
Response:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": """Failed to process request [{
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"forcemerge" : {
"max_num_segments" : 1
},
"rollover" : {
"max_age" : "1d",
"max_size" : "50gb"
},
"set_priority" : {
"priority" : 100
},
"shrink" : {
"number_of_shards" : 1
}
}
},
"delete" : {
"min_age" : "14d",
"actions" : {
"delete" : {
"delete_searchable_snapshot" : true
}
}
}
}
}
}] with errors: [[traces-apm.traces-default_policy] set as read-only by [file_settings]]"""
}
],
"type": "illegal_argument_exception",
"reason": """Failed to process request [{
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"forcemerge" : {
"max_num_segments" : 1
},
"rollover" : {
"max_age" : "1d",
"max_size" : "50gb"
},
"set_priority" : {
"priority" : 100
},
"shrink" : {
"number_of_shards" : 1
}
}
},
"delete" : {
"min_age" : "14d",
"actions" : {
"delete" : {
"delete_searchable_snapshot" : true
}
}
}
}
}
}] with errors: [[traces-apm.traces-default_policy] set as read-only by [file_settings]]"""
},
"status": 400
}
I removed the traces-apm.traces-default_policy from our stackConfigPolicy but it remains read-only.
How can we fix this without data loss?