Writing fields.yml

Elastic Stack Beats
1

Is there a documentation on how to write fields.yml when writing a new beat?

Especially some Elastic types seem to be ignored when generating index template.

Also how do you describe arrays?

Thanks,

2

@rvlander Great to hear that you are building a beat. A docs on how to create the beat.yml is currently missing. Did you start your beat based on beat-generator? https://github.com/elastic/beat-generator If yes you can use make update to generate the full configuration file out of beat.yml. For how to to arrays / list it's best to have a look at the existing configurations for example in filebeat: https://github.com/elastic/beats/blob/master/filebeat/filebeat.yml

Let me know if you have some more questions.

3

It's being my fourth beat!

@ruflin I started from topbeat as explained in the online documentation. I can already use make update since my Makefile hooks to libbeats scripts:

BEATNAME=mybeat
BEAT_DIR=somwhere/mybeat
ES_BEATS=$(GOPATH)/src/github.com/elastic/beats
GOPACKAGES=$(shell go list ${BEAT_DIR}/... | grep -v /vendor/)
include $(ES_BEATS)/libbeat/scripts/Makefile

I made a terrible mistake on my previous post, I was talking of fields.yml and not beat.yml.

There is also another strange thing: this is that when the beat publish events to elastic search, they seem no to use the index template generated on this purpose (manually posted in ElasticSearch).

4

Unfortunately the developer guide is a little bit out of date. It is now recommended to use the beat-generator as it includes already all the necessary files. The most recent version even has a first version of packing inside.

What is the index pattern you have defined in your template? Does it match with the index pattern used?

Please also feel free to post here the issues you were encountering when creating a beat as I plan to rewrite the developer guide soonish. I added fields.yml to my list.

In case your beats are public, don't forget to add them to our community beats list: https://github.com/elastic/beats/blob/master/libbeat/docs/communitybeats.asciidoc

5

Hi, what are the types supported in fields.yml? Specifically I'd like to dynamically nest data (e.g., use common.MapStr as value), is that supported? If not what's the recommended way to do it? With Logstash I get it for free with the Kafka input plugin, I was looking to achieve something equivalent natively in Beats.

I should add that my current version of fields.yml is a subset of what I'm sending and is resulting in this crash in elasticsearch 2.3.3

        at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$1.execute(MetaDataCreateIndexService.java:320)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Thanks!

6

I can acquire knowledge in this reply. Thank you.

กางเกงในซุปเปอร์แมน

8

Nested documents are possible. Best is to have a look at our existing fields.yml files for example here: https://github.com/elastic/beats/blob/master/metricbeat/etc/fields.yml Be aware that some tpyes are different between elasticsearch 2.x and 5.x

Can you share the fields.yml file that you are using?

9

Sorry I should have clarified that by "dynamically nest data" I meant that I don't know the structure ahead of time. It's basically a tree I build from notifications that I receive from another agent via updates/deletes. A notification looks something like this:

path: /foo
updates: bar=5

The simplest way to represent something like that in elasticsearch seems to be something like:

map[string]interface{}{
  "foo" : map[string]interface{}{"bar" : 5},
}

At least this approach worked pretty well with Logstash (no configuration required). Is there a way to do the same with Beats, without knowing anything about /foo/bar or anything else in the tree ahead of time? If not is there a way in Beats to handle this, besides sending the raw updates to Elasticsearch?

To answer your fields.yml question I don't have anything in there now, what I'd like to have is something like:

fields:
  - name: data
    type: <map?>
10

I assume what you are looking for are dynamic templates as we have here for example in packetbeat: https://github.com/elastic/beats/blob/master/packetbeat/packetbeat.template.json#L7 If you don't provide a template, elasticsearch just tries to guess. This works in most of the cases, but not all like if you send 0 first and it detects int, but it should be a float. Logstash also uses some generic default templates, so you could just use the one from logstash and change the index if that works for you.

There is a lot of magic that can be done with templates. For dynamic templates for example see here: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html Because we generate the templates with a script our script does not allow the full magic (yet). So manually creating the template is also always an option. Fields.yml is only here for our automation of docs and templates.

I hope this helps to bring you one step close to the solution.

11

Ok commenting out the template section of foobeat.yml worked. Not sure why just setting path: foobeat.template-es2x.json in it didn't work either. I even tried changing keyword to string in fields.yml but got the same exception.

Thanks for the help!

12

Could it be because the template was already applied before? If I understand you right, it is working now?

13

Yes, it's working after applying this diff:

index cb8e0bb..afca590 100644
--- a/openconfigbeat.yml
+++ b/openconfigbeat.yml
@@ -45,16 +45,16 @@ output:
     # A template is used to set the mapping in Elasticsearch
     # By default template loading is enabled and the template is loaded.
     # These settings can be adjusted to load your own template or overwrite existing ones
-    template:
+    #template:
 
       # Template name. By default the template name is openconfigbeat.
-      name: "openconfigbeat"
+      #name: "openconfigbeat"
 
       # Path to template file
-      path: "openconfigbeat.template.json"
+      #path: "openconfigbeat.template.json"
 
       # Overwrite existing template
-      overwrite: false
+      #overwrite: false
 
     # Optional HTTP Path
     #path: "/elasticsearch"

Changing path to use openconfigbeat.template-es2x.json sounded like a better solution but didn't work.