Understanding the role of fields.yml

[Martin_H_Andersen (Martin Andersen)]

My setup is this: filebeat -> logstash -> elasticsearch

I don't understand why filebeat is responsible for defining the fields mapping in elasticsearch.
Fields can be altered in logstash? Giving new names.

We have a very fixed set of structured data we are logging should I load the mapping directly into ES? and how would I do that?

Thanks

[pierhugues (Pier-Hugues Pellerin)]

Hello @Martin_H_Andersen,

In beats we try to provide an out of the box experience as much as possible in the scenario beats sending directly to Elasticsearch. Beats take care of any version upgrade, it make sure the index pattern is correct. The fields.yml is used to generate the elasticsearch JSON template.

Doing so allow us to create custom dashboard to visualize the information, because the fields will have the correct types.

When you are using Logstash, we consider this as a more advanced use case where user will either:

1. use the LS default dynamic template
2. create their own templates depending on the fields they have
3. will use a beat template as a starting point.

Concerning the point 3 and since you have custom fields you could use the command:

{beatname} export template > mytemplate.json

This will give you the template that the beat is using, you can modify it with your custom fields and push it to elasticsearch.

Do this answer your question?