# ECS fields not applied from index template **URL:** \ **Category:** Logstash\ **Created:** [March 12, 2021, 8:34am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056 "2021-03-12T08:34:39Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jfs1](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jfs1](https://discuss.elastic.co/u/jfs1)\ **Post date:** [March 12, 2021, 8:34am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/1 "2021-03-12T08:34:39Z")
I'm trying to reformat my index template to migrate to ECS. I think I'm missing something because the field format for the multi-level keys is not taken into account. My index template looks like this : ```auto { "template": { "settings": { "index": { "lifecycle": { "name": "fwlogs" }, "codec": "best_compression", "mapping": { "ignore_malformed": "true" }, "refresh_interval": "60s", "number_of_replicas": "0" } }, "mappings": { "dynamic_templates": [], "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "keyword", "index": false }, "event": { "properties": { "category": { "type": "keyword", "ignore_above": 1024 }, "kind": { "type": "keyword", "ignore_above": 1024 }, "module": { "type": "keyword", "ignore_above": 1024 } } }, "host": { "type": "ip" }, "message": { "type": "text" } } }, "aliases": {} } } ``` But my indices show "unknown field" for all multi-level fields : ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/4/a4c1926129e9160a3b9047e6ae11ea43bca6c335.png) I'm using this logstash filter to translate my fields : ```auto filter{ if [host] in ["...", "..."] { mutate { add_field => { "event.category" => "network" } add_field => { "event.kind" => "event" } add_field => { "event.module" => "juniper" } } grok { match => ["message", "%{TIMESTAMP_ISO8601:timestamp_tmp} - %{DATA:vpn} - \[%{IPV4:src_ip}\] %{DATA:src_user}\(%{WORD:domain}\)\[%{DATA:role}\] - %{GREEDYDATA:juni_message}" ] } if [juni_message] =~ /Key Exchange number [0-9]+ occurred for user with NCIP/ { grok { match => ["juni_message", "Key Exchange number %{NUMBER} occurred for user with NCIP %{IP:client.ip}"] } } else if [juni_message] =~ /is deleted since user does not qualify reevaluated policies/ { mutate { add_field => { "event.reason" => "User does not qualify reevaluated policies" } } } mutate { remove_field => ["juni_message"] } } } ``` I'm probably missing a flag or a small tip either in logstash or in the index template, but I can't figure out where. Any idea ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 12, 2021, 9:21am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/2 "2021-03-12T09:21:58Z")
Could you try this instead? ```auto add_field => { "[event][category]" => "network" } ``` ---
**Author:** ![jfs1](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jfs1](https://discuss.elastic.co/u/jfs1)\ **Post date:** [March 12, 2021, 10:19am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/3 "2021-03-12T10:19:09Z")
Did the change, restarted logstash. Same symptoms (but I'm still on the same index template). Should I try explicit nested fields in the template ? I.e. adding "type: nested" as in : ```auto "event": { "type": "nested", "properties": { "category": { ``` Or should I just rotate my indices ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 12, 2021, 10:31am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/4 "2021-03-12T10:31:01Z")
No. Don't use `nested` here. Did you remove the index first? If it still does not work, I'd recommend moving this discussion to #elastic-stack:logstash . ---
**Author:** ![jfs1](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jfs1](https://discuss.elastic.co/u/jfs1)\ **Post date:** [March 15, 2021, 10:26am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/5 "2021-03-15T10:26:26Z")
No. I can't remove the index as this is live data I can't afford to loose. I usually wait for the next log rotation at midnight to have the new template used. Indeed, the root cause seem to lie in the way logstash is configured. I have another issue there (ilm not applied), which may have the same root cause. BTW, how can I move this topic or add logstash as label ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 15, 2021, 11:27am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/6 "2021-03-15T11:27:35Z")
I moved the thread to #elastic-stack:logstash ---
**Author:** ![jfs1](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jfs1](https://discuss.elastic.co/u/jfs1)\ **Post date:** [March 25, 2021, 10:01am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/7 "2021-03-25T10:01:02Z")
OK. Found the culprit. My index template had fields that where conflicting with ECS structure. More specifically, I had " host" and a "url" keyword fields in the template. This conflict was not detected by ES when pushing the new index template. That's why I had this strange behaviour of "everything looks fine, but it fails anyway". I renamed the offending fields in the template and changed the logstash ingest logic and everything is on track now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 22, 2021, 10:01am UTC](https://discuss.elastic.co/t/ecs-fields-not-applied-from-index-template/267056/8 "2021-04-22T10:01:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.