# How to disable loading template for Elasticsearch 2.x? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 7, 2017, 4:57pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701 "2017-03-07T16:57:40Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\ **Post date:** [March 7, 2017, 4:57pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/1 "2017-03-07T16:57:40Z")
Hi everyone! @Update: Head to first comment to see my current problem I try to load JSON formatted logs from log file into elasticsearch. Everything works OK, but I have those json.message, json.@timestamp, json.whatever keys stored in elasticsearch. From what I've read the parameter: `json.keys_under_root: true` Handles this problem well, so I'd see normal fields. But when I add it to the yml file and restart filebeat service, filebeat can't ship logs to elasticsearch anymore. My config: ``` filebeat: prospectors: - paths: - /tmp/logs/logs.log json: message_key: message keys_under_root: true overwrite_keys: true output: elasticsearch: hosts: ["localhost:9200"] index: "logstash-%{+yyyy.MM.dd}" ``` Without keys\_under\_root everything works properly. Elasticsearch and Kibana are on docker containers and in version 5.2.0. Filebeat is 5.2.2. Another thing I'd like to know is: Can I get rid of all those Beats.Name, Beats.Hostname, Type fields? I can see, they are defined in filebeat.template.json and I'd like to get rid of them, but when I delete them and restart service, filebeat stops to ship logs to ES too. Could You please help me with those 2 matters or point me to right direction? Edit: To clarify, here is the example log: ![](https://us1.discourse-cdn.com/elastic/original/3X/3/3/336a6bfb2ce86119b3d87f69e57294e9f9879b13.png) What I want to achieve is to get rid of "json" prefix on custom fields and, if possible, I'd like to remove offset, source, type, input\_type and Beats.\* fields as I don't need them. Greetings, Wojtek ---
**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\ **Post date:** [March 8, 2017, 6:42pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/2 "2017-03-08T18:42:20Z")
I figured out the keys\_under\_root parameters. Now I have problem with elasticsearch output. These are the logs from filebeat: ``` 2017/03/10 10:30:08.451074 beat.go:267: INFO Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs] 2017/03/10 10:30:08.451099 beat.go:177: INFO Setup Beat: filebeat; Version: 5.2.2 2017/03/10 10:30:08.451108 processor.go:43: DBG Processors: 2017/03/10 10:30:08.451114 beat.go:183: DBG Initializing output plugins 2017/03/10 10:30:08.451171 logp.go:219: INFO Metrics logging every 30s 2017/03/10 10:30:08.451219 output.go:167: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json 2017/03/10 10:30:08.451334 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /usr/share/filebeat/bin/filebeat.template-es2x.json 2017/03/10 10:30:08.451388 outputs.go:100: ERR failed to initialize elasticsearch plugin as output: Error loading template /usr/share/filebeat/bin/filebeat.template-es2x.json: open /usr/share/filebeat/bin/filebeat.template-es2x.json: no such file or directory 2017/03/10 10:30:08.451401 beat.go:288: CRIT Exiting: error initializing publisher: Error loading template /usr/share/filebeat/bin/filebeat.template-es2x.json: open /usr/share/filebeat/bin/filebeat.template-es2x.json: no such file or directory Exiting: error initializing publisher: Error loading template /usr/share/filebeat/bin/filebeat.template-es2x.json: open /usr/share/filebeat/bin/filebeat.template-es2x.json: no such file or directory ``` The logs are very verbose, I really don't have this file there, but I don't want to use elasticsearch 2x, I just want 5.2. Is there a parameter to specify this in output section in filebeat.yml? My current config goes like this: ``` filebeat: prospectors: - paths: - /tmp/logs/logs.log json: message_key: message keys_under_root: true overwrite_keys: true output: elasticsearch: hosts: ["localhost:9200"] #index: "logstash-%{+yyyy.MM.dd}" template: enabled: true #name: "logstash" path: "/etc/filebeat/filebeat.template.json" overwrite: true ``` Cheers! ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [March 10, 2017, 3:21pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/3 "2017-03-10T15:21:26Z")
You can add `versions.2x.enabled: false` under the `template` section, see [this doc](https://www.elastic.co/guide/en/beats/filebeat/5.2/elasticsearch-output.html#_template_versions). The reason we check for the file even if you use ES 5.x is that at startup time we don't yet have a connection to ES, so we don't know it's version. Delaying the error would make it harder to notice (by then, the process is deamonized). ---
**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\ **Post date:** [March 10, 2017, 3:48pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/4 "2017-03-10T15:48:02Z")
Thank You very much for the response, that was exactly what I was looking for. Hovewer it seems that I try to import my custom template in a wrong way. I edited the `filebeat.template.json` file and still I get the default messages template. Is there some proper way to input the custom template into elasticsearch? I've used ``` curl -XPUT 'http://localhost:9200/_template/logstash' -d@/etc/filebeat/filebeat.template.json ``` But without luck. ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [March 10, 2017, 4:12pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/5 "2017-03-10T16:12:02Z")
What did you modify in the template? The index pattern? The curl command looks good, but should not be needed, Filebeat itself can also do it. Also, keep in mind that the new template only applies to new indices, not to the existing ones. ---
**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\ **Post date:** [March 10, 2017, 4:17pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/6 "2017-03-10T16:17:11Z")
I (don't know if it's correct) removed the beats.\* fields, source, type, offset etc fields that filebeat adds to message. I just need my custom fields, thus I wanted to get rid of those filebeat fields somehow. @Edit, hey, I'm doing something stupid here - I'm editing the filebeat.template.json in /etc/filebeat directory and now I have the default template there. I guess it is because I set overwrite to true and thus I do something ridiculous 😃 What I'd like to achieve is to have below template pattern sent to elasticsearch: ``` { "mappings": { "_default_": { "_all": { "norms": false }, "_meta": { "version": "5.2.2" }, "dynamic_templates": [{ "strings_as_keyword": { "mapping": { "ignore_above": 1024, "type": "keyword" }, "match_mapping_type": "string" } }], "properties": { "@timestamp": { "type": "date" }, "message": { "norms": false, "type": "text" }, "meta": { "properties": { "cloud": { "properties": { "availability_zone": { "ignore_above": 1024, "type": "keyword" }, "instance_id": { "ignore_above": 1024, "type": "keyword" }, "machine_type": { "ignore_above": 1024, "type": "keyword" }, "project_id": { "ignore_above": 1024, "type": "keyword" }, "provider": { "ignore_above": 1024, "type": "keyword" }, "region": { "ignore_above": 1024, "type": "keyword" } } } } } } } }, "order": 0, "settings": { "index.mapping.total_fields.limit": 10000, "index.refresh_interval": "5s" }, "template": "logstash-*" } ``` ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [March 10, 2017, 4:23pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/7 "2017-03-10T16:23:13Z")
> I (don't know if it's correct) removed the beats.\* fields, source, type, offset etc fields that filebeat adds to message. I just need my custom fields, thus I wanted to get rid of those filebeat fields somehow. That's not correct, removing them from the template will not going to remove it from the data, just let ES choose the types for them. See the `drop_fields` and `include_fields` processor for filtering out fields: [doc link](https://www.elastic.co/guide/en/beats/filebeat/5.2/configuration-processors.html) ---
**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\ **Post date:** [March 10, 2017, 4:25pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/8 "2017-03-10T16:25:30Z")
Is ithis preprocessing costly operation? I mean for CPU? I always prefer rather to send logs prepared than to operate on them and parse somehow. It's because there will be like 9,5 M of logs per hour, so, ouch... ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [March 10, 2017, 4:27pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/9 "2017-03-10T16:27:45Z")
Just dropping fields (without any actual parsing) is pretty cheap, and probably saves you CPU time on the JSON encoding, so you might get a net reduction in CPU. I'd say just try it. If not, you can also easily drop fields in Logstash or Elasticsearch Ingest Node. ---
**Author:** ![Wojter](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Wojter](https://discuss.elastic.co/u/Wojter)\ **Post date:** [March 10, 2017, 4:36pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/10 "2017-03-10T16:36:16Z")
@tudor, You're life-saver here, I managed to have the output I wanted with only my fields, metadata and timestamp thanks to You. It's a shame that I can't remove "type" field as specified in documentation, but it's not that much. Thanks a ton - I can now omit Logstash and send directly to elasticsearch using less CPU, memory etc with pure filebeat 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 7, 2017, 4:36pm UTC](https://discuss.elastic.co/t/how-to-disable-loading-template-for-elasticsearch-2-x/77701/11 "2017-04-07T16:36:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.