Using a single Life Cycle Policy for many Index Templates

[dturner]

Hello,

We are using Elastic agent with different policies for different integrations. This has created many index templates for these different data sources. Each one of these index templates is using a different index lifecycle policy. Instead of managing dozens of lifecycle policies, we would like to point the existing index templates to one of our custom index lifecycle policies. We would also like any new data source to use one of our custom index lifecycle policies. How can this be done? Is there a better way than manually adding each individual index template to a lifecycle policy?

I have added the primary logs index template ( logs--) to our custom lifecycle policy, however, this does not apply to all indicies as many are explicitly looking for an individual index source like
logs-cisco_ftd.log-* or logs-elastic_agent-*.

Thank you for your help!
Derrick

[rugenl (Len Rugen)]

If you are on a current version, you should be using compostable templates. It's not the template name that matches, but the index_patterns in the template. If multiple templates have matching index_patterns, the one with the highest priority will apply.

Check the patterns and priorities of your new templates. See the section about pattern collisions

[dturner]

Thank you very much. I attempted this however I received the error below. I did not want to have to create mappings and data streams for each of the indexes. I created a compostable template with the custom lifecycle policy and applied that to a index template with a priority of 500 to match log*.

(If the image doesn't load the error message is below)

composable template [test] with index patterns [logs*], priority [500] and no data stream configuration would cause data streams [logs-network_traffic.http-default, logs-network_traffic.flow-default, .fleet-actions-results, logs-mimecast.ttp_ap_logs-default, logs-elastic_agent.metricbeat-default, logs-endpoint.events.file-default, logs-network_traffic.tls-default, logs-network_traffic.icmp-default, logs-endpoint.events.process-default-2022.07.25-000001_reindex, logs-elastic_agent.osquerybeat-default, logs-system.system-default, logs-osquery_manager.result-default, logs-endpoint.events.network-default, logs-mimecast.ttp_ip_logs-default, logs-endpoint.events.network-default-2022.07.25-000001_reindex, logs-enterprise_search.audit-default, logs-elastic_agent.endpoint_security-default, logs-o365.audit-default, logs-mimecast.audit_events-default, logs-network_traffic.dhcpv4-default, logs-system.syslog-default, logs-endpoint.alerts-default, logs-windows.powershell_operational-default, logs-endpoint.events.security-default, logs-system.security-default, logs-system.security-default-2022.07.25-000001_reindex, logs-endpoint.events.process-default, logs-mimecast.siem_logs-default, logs-endpoint.events.registry-default, logs-elastic_agent.filebeat-default, logs-windows.sysmon_operational-default, logs-windows.powershell-default, logs-mimecast.ttp_url_logs-default, logs-network_traffic.dns-default, logs-system.application-default, logs-endpoint.events.file-default-2022.07.25-000001_reindex, logs-cisco_secure_endpoint.event-default, logs-elastic_agent-default, logs-mimecast.dlp_logs-default, logs-system.syslog-default-2022.07.25-000001_reindex, logs-endpoint.events.library-default, logs-system.auth-default, logs-elastic_agent.packetbeat-default, logs-enterprise_search.api-default] to no longer match a data stream template

[dturner]

We have 80 indexes from different data sources. Do we have to manually add the ILM compostable template to each one?

[dturner]

We have been manually adding the composable template to each individual index template. Is there a better way? This is not a good way to manage it.