Only able to setup Metricbeat with ILM and a custom index template going against the documentation

Elastic Stack Beats
,
1

Hi,
I'll try to be succint, but will provide context if anyone is interested!

Currently running ELK v8.17.1 and fiddling with Beats using the same version.

According to the index section for the metricbeat configuration file
"Custom index settings are ignored when ILM is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Index lifecycle management (ILM) to learn how to change the index name."

Similarly, for indices
"Similar to index, defining custom indices will disable Index lifecycle management (ILM)."

However, the custom index only starts receving data with the following configuration

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.template.name: "metricbeat-docker-%{[agent.version]}"
setup.template.pattern: "metricbeat-docker-%{[agent.version]}"

tags: ["docker"]

output.elasticsearch:
  type: elasticsearch
  hosts: ["es.ip.add.ress:9200"]
  preset: balanced
  protocol: "https"
  api_key: "something:something"
  ssl:
    certificate_authorities: | 
      -----BEGIN CERTIFICATE-----
      MIIDSjCCAjKgAwI (and so on)                                   
      -----END CERTIFICATE-----
    verification_mode: certificate
  index: "metricbeat-docker-%{[agent.version]}" # this line

setup.ilm.enabled: true
setup.ilm.policy_name: "metricbeat-docker"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Note the index line in the output.elasticsearch section. So I'm not sure what I'm missing.
EDIT: the datastream works and rolls over as expected according to the ILM (simple test with 1 day Hot tier and deletion after 2 days).

Furthermore, since an ILM policy can be shared between several index templates, where one defines the index patterns, how would only mentioning to the ILM policy in the Beats YAML file allow for custom indices? Could someone point out what I am misinterpreting?

Is it because setup.ilm.rollover_alias, setup.ilm.pattern that I've seen mentioned in suggested posts were seemingly removed without the documentation entirely reflecting the change?

Thanks!

P.S. I am noticing a push towards the Elastic Agent, are Beats on their way out?