# Writing fields.yml **URL:** \ **Category:** Beats\ **Created:** [March 14, 2016, 3:12pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336 "2016-03-14T15:12:20Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![rvlander](https://avatars.discourse-cdn.com/v4/letter/r/a6a055/32.png) [@rvlander](https://discuss.elastic.co/u/rvlander)\ **Post date:** [March 14, 2016, 3:12pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/1 "2016-03-14T15:12:20Z")
Is there a documentation on how to write fields.yml when writing a new beat? Especially some Elastic types seem to be ignored when generating index template. Also how do you describe arrays? Thanks, ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 14, 2016, 3:33pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/2 "2016-03-14T15:33:05Z")
@rvlander Great to hear that you are building a beat. A docs on how to create the beat.yml is currently missing. Did you start your beat based on beat-generator? [https://github.com/elastic/beat-generator](https://github.com/elastic/beat-generator) If yes you can use `make update` to generate the full configuration file out of beat.yml. For how to to arrays / list it's best to have a look at the existing configurations for example in filebeat: [https://github.com/elastic/beats/blob/master/filebeat/filebeat.yml](https://github.com/elastic/beats/blob/master/filebeat/filebeat.yml) Let me know if you have some more questions. ---
**Author:** ![rvlander](https://avatars.discourse-cdn.com/v4/letter/r/a6a055/32.png) [@rvlander](https://discuss.elastic.co/u/rvlander)\ **Post date:** [March 14, 2016, 4:37pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/3 "2016-03-14T16:37:19Z")
It's being my fourth beat! @ruflin I started from topbeat as explained in the online documentation. I can already use make update since my Makefile hooks to libbeats scripts: ```auto BEATNAME=mybeat BEAT_DIR=somwhere/mybeat ES_BEATS=$(GOPATH)/src/github.com/elastic/beats GOPACKAGES=$(shell go list ${BEAT_DIR}/... | grep -v /vendor/) include $(ES_BEATS)/libbeat/scripts/Makefile ``` I made a terrible mistake on my previous post, I was talking of **fields.yml** and not beat.yml. There is also another strange thing: this is that when the beat publish events to elastic search, they seem no to use the index template generated on this purpose (manually posted in ElasticSearch). ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 18, 2016, 8:17am UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/4 "2016-03-18T08:17:31Z")
Unfortunately the developer guide is a little bit out of date. It is now recommended to use the beat-generator as it includes already all the necessary files. The most recent version even has a first version of packing inside. What is the index pattern you have defined in your template? Does it match with the index pattern used? Please also feel free to post here the issues you were encountering when creating a beat as I plan to rewrite the developer guide soonish. I added fields.yml to my list. In case your beats are public, don't forget to add them to our community beats list: [https://github.com/elastic/beats/blob/master/libbeat/docs/communitybeats.asciidoc](https://github.com/elastic/beats/blob/master/libbeat/docs/communitybeats.asciidoc) ---
**Author:** ![7AC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/7ac/32/10139_2.png) [@7AC](https://discuss.elastic.co/u/7AC)\ **Post date:** [June 3, 2016, 9:37pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/5 "2016-06-03T21:37:50Z")
Hi, what are the types supported in fields.yml? Specifically I'd like to dynamically nest data (e.g., use common.MapStr as value), is that supported? If not what's the recommended way to do it? With Logstash I get it for free with the Kafka input plugin, I was looking to achieve something equivalent natively in Beats. I should add that my current version of fields.yml is a subset of what I'm sending and is resulting in this crash in elasticsearch 2.3.3 ```auto at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:320) at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ``` Thanks! ---
**Author:** ![sittideat\_kitporkha](https://avatars.discourse-cdn.com/v4/letter/s/22d042/32.png) [@sittideat\_kitporkha](https://discuss.elastic.co/u/sittideat_kitporkha)\ **Post date:** [June 4, 2016, 1:16pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/6 "2016-06-04T13:16:06Z")
I can acquire knowledge in this reply. Thank you. [**กางเกงในซุปเปอร์แมน**](http://supermangodlike.wordpress.com) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 6, 2016, 1:03pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/8 "2016-06-06T13:03:28Z")
Nested documents are possible. Best is to have a look at our existing fields.yml files for example here: [https://github.com/elastic/beats/blob/master/metricbeat/etc/fields.yml](https://github.com/elastic/beats/blob/master/metricbeat/etc/fields.yml) Be aware that some tpyes are different between elasticsearch 2.x and 5.x Can you share the fields.yml file that you are using? ---
**Author:** ![7AC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/7ac/32/10139_2.png) [@7AC](https://discuss.elastic.co/u/7AC)\ **Post date:** [June 6, 2016, 5:44pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/9 "2016-06-06T17:44:13Z")
Sorry I should have clarified that by "dynamically nest data" I meant that I don't know the structure ahead of time. It's basically a tree I build from notifications that I receive from another agent via updates/deletes. A notification looks something like this: ```auto path: /foo updates: bar=5 ``` The simplest way to represent something like that in elasticsearch seems to be something like: ```auto map[string]interface{}{ "foo" : map[string]interface{}{"bar" : 5}, } ``` At least this approach worked pretty well with Logstash (no configuration required). Is there a way to do the same with Beats, without knowing anything about `/foo/bar` or anything else in the tree ahead of time? If not is there a way in Beats to handle this, besides sending the raw updates to Elasticsearch? To answer your `fields.yml` question I don't have anything in there now, what I'd like to have is something like: ```auto fields: - name: data type: ``` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 7, 2016, 12:57pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/10 "2016-06-07T12:57:41Z")
I assume what you are looking for are dynamic templates as we have here for example in packetbeat: [https://github.com/elastic/beats/blob/master/packetbeat/packetbeat.template.json#L7](https://github.com/elastic/beats/blob/master/packetbeat/packetbeat.template.json#L7) If you don't provide a template, elasticsearch just tries to guess. This works in most of the cases, but not all like if you send 0 first and it detects int, but it should be a float. Logstash also uses some generic default templates, so you could just use the one from logstash and change the index if that works for you. There is a lot of magic that can be done with templates. For dynamic templates for example see here: [https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html) Because we generate the templates with a script our script does not allow the full magic (yet). So manually creating the template is also always an option. Fields.yml is only here for our automation of docs and templates. I hope this helps to bring you one step close to the solution. ---
**Author:** ![7AC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/7ac/32/10139_2.png) [@7AC](https://discuss.elastic.co/u/7AC)\ **Post date:** [June 10, 2016, 6:33pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/11 "2016-06-10T18:33:13Z")
Ok commenting out the `template` section of `foobeat.yml` worked. Not sure why just setting `path: foobeat.template-es2x.json` in it didn't work either. I even tried changing `keyword` to `string` in fields.yml but got the same exception. Thanks for the help! ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 13, 2016, 7:42am UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/12 "2016-06-13T07:42:52Z")
Could it be because the template was already applied before? If I understand you right, it is working now? ---
**Author:** ![7AC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/7ac/32/10139_2.png) [@7AC](https://discuss.elastic.co/u/7AC)\ **Post date:** [June 13, 2016, 4:41pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/13 "2016-06-13T16:41:20Z")
Yes, it's working after applying this diff: ```auto index cb8e0bb..afca590 100644 --- a/openconfigbeat.yml +++ b/openconfigbeat.yml @@ -45,16 +45,16 @@ output: # A template is used to set the mapping in Elasticsearch # By default template loading is enabled and the template is loaded. # These settings can be adjusted to load your own template or overwrite existing ones - template: + #template: # Template name. By default the template name is openconfigbeat. - name: "openconfigbeat" + #name: "openconfigbeat" # Path to template file - path: "openconfigbeat.template.json" + #path: "openconfigbeat.template.json" # Overwrite existing template - overwrite: false + #overwrite: false # Optional HTTP Path #path: "/elasticsearch" ``` Changing `path` to use `openconfigbeat.template-es2x.json` sounded like a better solution but didn't work. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:51pm UTC](https://discuss.elastic.co/t/writing-fields-yml/44336/14 "2017-07-05T21:51:12Z")