Dynamic value substitution for date is NULL

To quote the docs:

Writing to different indices: best practices
You cannot use dynamic variable substitution when ilm_enabled is true and when using ilm_rollover_alias.

So what I think I want to do is make each new batch of documents be ingested in a new index, where the name starts the same. Eg:


The example output config of

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

worked at first, creating "logstash-2021.12.10-000001". Im not 100% sure where the "-000001" came from, but that does not bug me too much.

but this no longer works. It creates the index of a literal "%{[@metadata][beat]}-%{[@metadata][version]}-".

Notice that the date part is entirely missing.

I found the able about ilm, and attempted to turn it off with:

POST /_ilm/stop

So... first: dont I need dynamic index names for ilm? And what setting is preventing this?

OK, I figured out the missing date part... My filter included:

remove_field => ["message", "@timestamp", "@version", "host"]

I was trying to have 'clean data'. lol!

Ill leave this here incase someone else needs to see my error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.