Dynamic value substitution for date is NULL

Elastic Stack Logstash
1

To quote the docs:

Writing to different indices: best practices
You cannot use dynamic variable substitution when ilm_enabled is true and when using ilm_rollover_alias.

So what I think I want to do is make each new batch of documents be ingested in a new index, where the name starts the same. Eg:

logstash-2021.12.10
logstash-2021.12.11
logstash-2021.12.12

The example output config of

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

worked at first, creating "logstash-2021.12.10-000001". Im not 100% sure where the "-000001" came from, but that does not bug me too much.

but this no longer works. It creates the index of a literal "%{[@metadata][beat]}-%{[@metadata][version]}-".

Notice that the date part is entirely missing.

I found the able about ilm, and attempted to turn it off with:

POST /_ilm/stop

So... first: dont I need dynamic index names for ilm? And what setting is preventing this?

2

OK, I figured out the missing date part... My filter included:

remove_field => ["message", "@timestamp", "@version", "host"]

I was trying to have 'clean data'. lol!

Ill leave this here incase someone else needs to see my error.