Index template order fields seems not to work

[Adriann (Adrian)]

Hello,

I have few templates each for every index name based on the module name for ilm management.
I notice that after the update to 7.12.1 all my indices have applied templates with order value 10 event though there are other templates with specific index patterns. What do I need to do to allow template filebeat-7.12.1 to exist to match default index patterns and to let specific index patterns matches with specific index template?

[warkolm (Mark Walkom)]

Are you referring specifically to lifecycle policy that points to filebeat in that last image?

[Adriann (Adrian)]

Yes, this is the lifecycle policy that is set up for the default index template pattern and this is the template that is applied to every index even though the merge number is set up to 10 and the other templates with more specific index pattern have set up the merge number to 1.

[warkolm (Mark Walkom)]

Do you mind pulling the actual json for that that template and posting it?

[Adriann (Adrian)]

Those are the setting of the template. It's a default template exported from filebeat(7.12.1). The only change was the index pattern, merge value, name and ilm policy. I cut the fields because of the letters limit.

{
"index": {
"lifecycle": {
"name": "filebeat-fortinet_policy"
},
"mapping": {
"total_fields": {
"limit": "10000"
}
},
"refresh_interval": "5s",
"number_of_shards": "1",
"final_pipeline": "fortinet-default-pipeline",
"max_docvalue_fields_search": "200",
"query": {
"default_field": [
"message",
"tags",
"agent.ephemeral_id",
"agent.id",
"agent.name",
"agent.type",
"agent.version",
"as.organization.name",
"client.address",
"client.as.organization.name",
"client.domain",
"client.geo.city_name",
"client.geo.continent_name",
"client.geo.country_iso_code",
"client.geo.country_name",
"client.geo.name",
"client.geo.region_iso_code",
"client.geo.region_name",
"client.mac",
"client.registered_domain",
"client.top_level_domain",
"client.user.domain",
"client.user.email",
"client.user.full_name",
"client.user.group.domain",
"client.user.group.id",
"client.user.group.name",
"client.user.hash",
"client.user.id",
"client.user.name",
"cloud.account.id",
"cloud.availability_zone",
"cloud.instance.id",
"cloud.instance.name",
"cloud.machine.type",
"cloud.provider",
"cloud.region",
"container.id",
"container.image.name",
"container.image.tag",
"container.name",
"container.runtime",
"destination.address",
"destination.as.organization.name",
"destination.domain",
"destination.geo.city_name",
"destination.geo.continent_name",
"destination.geo.country_iso_code",
"destination.geo.country_name",
"destination.geo.name",
"destination.geo.region_iso_code",
"destination.geo.region_name",
"destination.mac",
"destination.registered_domain",
"destination.top_level_domain",
"destination.user.domain",
"destination.user.email",
"destination.user.full_name",
"destination.user.group.domain",
"destination.user.group.id",
"destination.user.group.name",
"destination.user.hash",
"destination.user.id",
"destination.user.name",
"dns.answers.class",
"dns.answers.data",
"dns.answers.name",
"dns.answers.type",
"dns.header_flags",
"dns.id",
"dns.op_code",
"dns.question.class",
"dns.question.name",
"dns.question.registered_domain",
"dns.question.subdomain",
"dns.question.top_level_domain",
"dns.question.type",
"dns.response_code",
"dns.type",
"ecs.version",
"error.code",
"error.id",
"error.message",
"error.type",
"event.action",
"event.category",
"event.code",
"event.dataset",
"event.hash",
"event.id",
"event.kind",
"event.module",
"event.outcome",
"event.provider",
"event.timezone",
"event.type",
"file.device",
"file.directory",
"file.extension",
"file.gid",
"file.group",
"file.hash.md5",
"file.hash.sha1",
"file.hash.sha256",
"file.hash.sha512",
"file.inode",
"file.mode",
"file.name",
"file.owner",
"file.path",
"file.target_path",
"file.type",
"file.uid",
"geo.city_name",
"geo.continent_name",
"geo.country_iso_code",
"geo.country_name",
"geo.name",
"geo.region_iso_code",
"geo.region_name",
"group.domain",
"group.id",
"group.name",
"hash.md5",
"hash.sha1",
"hash.sha256",
"hash.sha512",
"host.architecture",
"host.geo.city_name",
"host.geo.continent_name",
"host.geo.country_iso_code",
"host.geo.country_name",
"host.geo.name",
"host.geo.region_iso_code",
"host.geo.region_name",
"host.hostname",
"host.id",
"host.mac",
"host.name",
"host.os.family",
"host.os.full",
"host.os.kernel",
"host.os.name",
"host.os.platform",
"host.os.version",
"host.type",
"host.user.domain",
"host.user.email",
"host.user.full_name",
"host.user.group.domain",
"host.user.group.id",
"host.user.group.name",
"host.user.hash",
"host.user.id",
"host.user.name",
"http.request.body.content",
"http.request.method",
"http.request.referrer",
"http.response.body.content",
"http.version",
"log.level",
"log.logger",
"log.origin.file.name",
"log.origin.function",
"log.syslog.facility.name",
"log.syslog.severity.name",
"network.application",
"network.community_id",
"network.direction",
"network.iana_number",
"network.name",
"network.protocol",
"network.transport",
"network.type",
"observer.geo.city_name",
"observer.geo.continent_name",
"observer.geo.country_iso_code",
"observer.geo.country_name",
"observer.geo.name",
"observer.geo.region_iso_code",
"observer.geo.region_name",
"observer.hostname",
"observer.mac",
"observer.name",
"observer.os.family",
"observer.os.full",
"observer.os.kernel",
"observer.os.name",
"observer.os.platform",
"observer.os.version",
"observer.product",
"observer.serial_number",
"observer.type",
"observer.vendor",
"observer.version",
"organization.id",
"organization.name",
"os.family",
"os.full",
"os.kernel",
"os.name",
"os.platform",
"os.version",
"package.architecture",
"package.checksum",
"package.description",
"package.install_scope",
"package.license",
"package.name",
"package.path",
"package.version",
"process.args",
"text",
"process.executable",
"process.hash.md5",
"process.hash.sha1",
"process.hash.sha256",
"process.hash.sha512",
"process.name",
"text",
"text",
"text",
"text",
"text",
"process.thread.name",
"process.title",
"process.working_directory",
"server.address",
"server.as.organization.name",
"server.domain",
"server.geo.city_name",
"server.geo.continent_name",
"server.geo.country_iso_code",
"server.geo.country_name",
"server.geo.name",
"server.geo.region_iso_code",
"server.geo.region_name",
"server.mac",
"server.registered_domain",
"server.top_level_domain",
"server.user.domain",
"server.user.email",
"server.user.full_name",
"server.user.group.domain",
"server.user.group.id",
"server.user.group.name",
"server.user.hash",
"server.user.id",
"server.user.name",
"service.ephemeral_id",
"service.id",
"service.name",
"service.node.name",
"service.state",
"service.type",
"service.version",
"source.address",
"source.as.organization.name",
"source.domain",
"source.geo.city_name",
"source.geo.continent_name",
"source.geo.country_iso_code",
"source.geo.country_name",
"source.geo.name",
"source.geo.region_iso_code",
"source.geo.region_name",
"source.mac",
"source.registered_domain",
"source.top_level_domain",
"source.user.domain",
"source.user.email",
"source.user.full_name",
"source.user.group.domain",
"source.user.group.id",
"source.user.group.name",
"source.user.hash",
"source.user.id",
"source.user.name",
"threat.framework",
"threat.tactic.id",
"threat.tactic.name",
"threat.tactic.reference",
"threat.technique.id",
"threat.technique.name",
"threat.technique.reference",
"text",
"trace.id",
"transaction.id",
"url.domain",
"url.extension",
"url.fragment",
"url.full",
"url.original",
"url.password",
"url.path",
"url.query",
"url.registered_domain",
"url.scheme",
"url.top_level_domain",
"url.username",
"text",
"text",
"user.domain",
"text",
"text",
"user.email",
"user.full_name",
"user.group.domain",
"user.group.id",
"user.group.name",
"user.hash",
"user.id",
"user.name",
"text",
"text",
"user_agent.device.name",
"user_agent.name",
"text",
"user_agent.original",
"user_agent.os.family",
"user_agent.os.full",
"user_agent.os.kernel",
"user_agent.os.name",
"user_agent.os.platform",
"user_agent.os.version",
"user_agent.version",
"text",
"agent.hostname",
"timeseries.instance",
"cloud.image.id",
"host.os.build",
"host.os.codename",
"kubernetes.pod.name",
"kubernetes.pod.uid",
"kubernetes.namespace",
"kubernetes.node.name",
"kubernetes.node.hostname",
"kubernetes.replicaset.name",
"kubernetes.deployment.name",
"kubernetes.statefulset.name",
"kubernetes.container.name",
"kubernetes.container.image",
"jolokia.agent.version",
"jolokia.agent.id",
"jolokia.server.product",
"jolokia.server.version",
"jolokia.server.vendor",
"jolokia.url",
"log.source.address",
"stream",
"input.type",
"syslog.severity_label",
"syslog.facility_label",
"process.program",
"log.flags",
"user_agent.os.full_name",
"fileset.name",
"icmp.code",
"icmp.type",
"igmp.type",
"azure.eventhub",
"azure.consumer_group",
"kafka.topic",
"kafka.key",
"activemq.caller",
"activemq.thread",
"activemq.user",
"activemq.log.stack_trace",
"apache.access.ssl.protocol",
"apache.access.ssl.cipher",
"apache.error.module",
"user.terminal",
"user.audit.id",
"user.audit.name",
"user.audit.group.id",
"user.audit.group.name",
"user.filesystem.id",
"user.filesystem.name",
"user.saved.id",
"user.saved.name",
"user.saved.group.id",
"user.saved.group.name",
"auditd.log.old_auid",
"auditd.log.new_auid",
"auditd.log.old_ses",
"auditd.log.new_ses",
"auditd.log.items",
"auditd.log.item",
"auditd.log.tty",
"auditd.log.a0",
"bucket_name",
"object_key",
"azure.subscription_id",
"azure.correlation_id",
"azure.tenant_id",
"zeek.notice.sub",
"zeek.notice.peer_name",
"zeek.notice.peer_descr",
"zeek.notice.actions",
"zeek.notice.email_body_sections",
"zeek.notice.email_delay_tokens",
"zeek.notice.identifier",
"fields.*"
]
},
"number_of_replicas": "0"
}
}

[Adriann (Adrian)]

When I deleted the template filebeat-7.12.1(red underline) no new index could be created because the filebeat user does not have the right to put new templates. While no new logs could be index I could see in filebeat logs

I have no idea who these index patterns do not want to work.

[Adriann (Adrian)]

I have the answer. It was the "." in the pattern. After changing to template index pattern from " filebeat-7.12.1-system-*" to "filebeat-*-system-*" It started to work properly.