# Logstash loading template, but not applying it? **URL:** \ **Category:** Logstash\ **Created:** [June 1, 2020, 4:55pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207 "2020-06-01T16:55:53Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![AndyS](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AndyS](https://discuss.elastic.co/u/AndyS)\ **Post date:** [June 1, 2020, 4:55pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/1 "2020-06-01T16:55:54Z")
I'm having a problem where logstash is loading a template file, but either it's not applying it, or this isn't being communicated to Elastic. (This is using Security Onion, which has logstash in Docker) I have a logstash filter that adds a field "network\_name", this is visibile in Kibana but as it's not an idex I can't use it for filters. My tests from within Docker... ### Check the template file is binding correctly and being used by Docker ```auto $ ls -lt /logstash-template.json -rw-r--r-- 1 root root 77692 Jun 1 16:21 /logstash-template.json ``` ### Check this file has my config: ```auto bash-4.2$ grep "network_name" /logstash-template.json -A 6 "network_name":{ "type":"text", "fields":{ "keyword":{ "type":"keyword" } } ``` ### Check the file is being loaded ```auto bash-4.2$ grep "logstash-template" /var/log/logstash/logstash.log | tail -n 5 [2020-06-01T15:38:16,981][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"} ``` ### Check the file has the right contents (positive, "network\_name" exists near the end) ```auto [2020-06-01T16:28:32,641][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"} [2020-06-01T16:28:32,646][INFO][logstash.outputs.elasticsearch] e"=>"keywo [..removed...] pe"=>"ip"}, "network_name"=>{"type"=>"text", "fields"=>{"keyword"=>{"type"=>"keyword"}}}, "year"=>{"type"=>"long"}}}}}} ``` ### "z" is a defaultfield, as a second test I removed it ```auto bash-4.2$ grep "\"z\"" /logstash-template.json | wc -l 0 ``` However it does not appear this file that is loaded is applied. In Kibana when I: management -\> index patterns -\> reindex ....z is still listed, network\_name is not Can anyone add advice please. Is it possible/how do I test deeper if the updated template file is being applied Are any changes needed pushing to Elastic to honour this change? Many thanks Andy ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 1, 2020, 5:08pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/2 "2020-06-01T17:08:18Z")
Use the [mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html) to ask elasticsearch what the mapping for the index is. > as it's not an idex I can't use it for filters It is unclear what you mean by this. ---
**Author:** ![AndyS](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AndyS](https://discuss.elastic.co/u/AndyS)\ **Post date:** [June 1, 2020, 9:43pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/3 "2020-06-01T21:43:43Z")
Thanks @Badger ... In Kibana, one log entry where the tags and field is applied has these tags ```auto tags syslogng, bro, dns, external_destination, internal_source, home_network ``` The only output specified for "bro in tags" ```auto output { if "bro" in [tags] and "test_data" not in [tags] and "import" not in [tags] { # stdout { codec => rubydebug } elasticsearch { hosts => elasticsearch index => "logstash-bro-%{+YYYY.MM.dd}" template_name => "logstash" template => "/logstash-template.json" template_overwrite => true } } } ``` Then looking at the mapping in Elastic ```auto GET /*logstash*/_mapping ``` ...z exists, "network\_name" does not ```auto "z" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword" } } } ``` From looking at docker/logstash/elastic config I don't see how mappings/logstash definitions are provided to elastic. Ignoring this setup, which part of the config in Elastic defines this? "It is unclear what you mean by this." Sorry, I mean this ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8fa9810300072d7424508aa11bd3b234e4f81803.png) "network\_name" exists, but no mapping. I followed the instructions to refresh on index but it never adds "network\_name" or removes "z" ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/c/3c76dcfa4c46ffe5be956d38249ad66a5c976e6e.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 1, 2020, 10:03pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/4 "2020-06-01T22:03:36Z")
I cannot answer the kibana question. In your template, what is the value of index\_patterns? Are you sure the template actually applies to the index you are writing to? ---
**Author:** ![AndyS](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AndyS](https://discuss.elastic.co/u/AndyS)\ **Post date:** [June 1, 2020, 10:34pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/5 "2020-06-01T22:34:24Z")
I know what you mean it feels like "logstash-template.json" isn't get imported, but from the logs I can see that it is being read ```auto [2020-06-01T15:38:16,981][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/logstash-template.json"} ``` This config applies to "logstash-bro-\*" ```auto # grep "index_patterns" logstash-template.json "index_patterns": ["logstash-ids-*", "logstash-firewall-*", "logstash-syslog-*", "logstash-bro-*", "logstash-import-*", "logstash-beats-*"], ``` The output for "bro" inputs ```auto index => "logstash-bro-%{+YYYY.MM.dd}" ``` Here is the full entry from Elastic. It has the tags (bro) and it has the right index, but not "network\_name" as a field ```auto { "took" : 30, "timed_out" : false, "_shards" : { "total" : 120, "successful" : 120, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 2.0, "hits" : [ { "_index" : "logstash-bro-2020.06.01", "_type" : "doc", "_id" : "3aTPcXIBGLNnyZGQ0wPh", "_score" : 2.0, "_source" : { "answers" : [ "87.194.89.8", "87.194.8.8", "87.194.9.8" ], "destination_ips" : "8.8.4.4", "source_ip" : "192.168.1.3", "protocol" : "udp", "destination_ip" : "8.8.4.4", "event_type" : "bro_dns", "parent_domain_length" : 11, "syslog-facility" : "user", "host" : "gateway", "query_class" : 1, "aa" : false, "transaction_id" : 5364, "syslog-priority" : "notice", "query" : "epdg.epc.mnc010.mcc234.pub.3gppnetwork.org", "network_name" : "home_network", "rcode" : 0, "query_type" : 1, "creation_date" : "2002-05-15T13:35:38.000Z", "ips" : [ "192.168.1.3", "8.8.4.4" ], "subdomain_frequency_score" : 2.16, "syslog-host" : "seconion-NU691", "ra" : true, "tags" : [ "syslogng", "bro", "dns", "external_destination", "internal_source", "home_network" ], "ttls" : [ 95.0, 95.0, 95.0 ], "rd" : true, "port" : 55732, "frequency_scores" : [ "3.9616", "3.8966", "2.16" ], "subdomain" : "epdg.epc.mnc010.mcc234.pub", "parent_domain_frequency_score" : 3.8966, "syslog-tags" : ".source.s_bro_dns", "syslog-host_from" : "seconion-nu691", "parent_domain" : "3gppnetwork", "syslog-sourceip" : "127.0.0.1", "query_class_name" : "C_INTERNET", "highest_registered_domain" : "3gppnetwork.org", "destination_port" : 53, "rejected" : false, "top_level_domain" : "org", "source_ips" : "192.168.1.3", "uid" : "CPd9873rVfIU2iH4Bf", "highest_registered_domain_frequency_score" : 3.9616, "destination_geo" : { "ip" : "8.8.4.4", "latitude" : 37.751, "country_name" : "United States", "country_code2" : "US", "continent_code" : "NA", "country_code3" : "US", "location" : { "lon" : -97.822, "lat" : 37.751 }, "longitude" : -97.822 }, "source_port" : 61982, "syslog-file_name" : "/nsm/bro/logs/current/dns.log", "@version" : "1", "timestamp" : "2020-06-01T21:35:44.996Z", "logstash_time" : 0.022396087646484375, "message" : """{"ts":"2020-06-01T21:35:43.991106Z","uid":"CPd9873rVfIU2iH4Bf","id.orig_h":"192.168.1.3","id.orig_p":61982,"id.resp_h":"8.8.4.4","id.resp_p":53,"proto":"udp","trans_id":5364,"rtt":0.008337974548339844,"query":"epdg.epc.mnc010.mcc234.pub.3gppnetwork.org","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["87.194.89.8","87.194.8.8","87.194.9.8"],"TTLs":[95.0,95.0,95.0],"rejected":false}""", "tld" : { "subdomain" : "epdg.epc.mnc010.mcc234.pub.3gppnetwork.org" }, "subdomain_length" : 26, "rcode_name" : "NOERROR", "tc" : "false", "site" : "0", "query_length" : 42, "@timestamp" : "2020-06-01T21:35:43.991Z", "rtt" : 0.008337974548339844, "query_type_name" : "A", "z" : 0 } } ] } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 1, 2020, 10:39pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/6 "2020-06-01T22:39:28Z")
> [@AndyS](#): > > ` "network_name" : "home_network",` In what sense does that elasticsearch record not have a network\_name field? 😃 ---
**Author:** ![AndyS](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AndyS](https://discuss.elastic.co/u/AndyS)\ **Post date:** [June 1, 2020, 11:07pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/7 "2020-06-01T23:07:17Z")
Yes, that's the frustration lol 🙂 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8fa9810300072d7424508aa11bd3b234e4f81803.png) (sorry, back to Kibana lol) In the image the field name exists, but the "!" (and the greyed out +/- signs) means it's not cached, so I can't filter on it. Do you think this is an Elastic/Kibana question then? (i.e. separate forum) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 1, 2020, 11:08pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/8 "2020-06-01T23:08:34Z")
I would try asking in the kibana forum. ---
**Author:** ![AndyS](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@AndyS](https://discuss.elastic.co/u/AndyS)\ **Post date:** [June 1, 2020, 11:14pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/9 "2020-06-01T23:14:45Z")
Will do, thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 29, 2020, 11:16pm UTC](https://discuss.elastic.co/t/logstash-loading-template-but-not-applying-it/235207/10 "2020-06-29T23:16:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.