Policies gets updated itself own its own, abruptly triggering rollover of indexes managed by default policies. As a result number of shards are reaching maximum limit in cluster.
Giving Errors like
Index lifecycle error
validation_exception: Validation Failed: 1: this action would add [2] shards, but this cluster currently has [2999]/[3000] maximum normal shards open;
{"log.level":"error","@timestamp":"2024-03-05T16:59:15.884+0530","log.origin":{"file.name":"go-docappender@v0.1.0/appender.go","file.line":279},"message":"failed to index document (illegal_argument_exception): Number of documents in the index can't exceed [2147483519]","service.name":"apm-server","ecs.version":"1.6.0"}
**If anyone Knows why these abrupt rollovers are happening, same thing has occurred 3 days ago also and why are ILMs updating themselves. **
Kibana version: 8.8.1
Elasticsearch version: 8.8.0
APM Server version: 8.8.1
we are using self managed binary APM server set up