Without any configuration changes, the elastic cloudwatch snapshot runs every 5 minutes (event though the frequency is set to 30m). Despite this frequent running, monitoring says the cluster is "healthy with warnings" due to "last successful backup was 18 hours ago". Finally, we are set to retain 100 snapshots, but kibana shows approximately 560 snapshots.
It seems like kibana is repeatedly trying to take snapshots, failing, and restarting. Suspect this is some sort of timeout because we do not have the same error when frequency is set to 4h.
There are a few changes which may or may not be related to this snapshot failure. We started noticing these changes after maxing out on the shard count (ie 2000 shards). we deleted old indices and attached ilm policies to index templates.. We also upgraded the version of functionbeat we are running to version 7.8.
We are running elastic 7.4.2.
What is causing this failure? What do we need to do to retain 30m snapshots?