Beats fields.yml and multi-fields

Elastic Stack Beats
1

Hello,

I really appreciate the great work on the beats. But the documentation for the new fields.yml is just awful or plainly missing :wink:
Can someone explain to me how to achieve multi-field mapping as described here:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/multi-fields.html
within fields.yml ?

Many thanks and best regards,

Sirk

2

Missing is probably the better word for it :frowning:

Here is an example: https://github.com/elastic/beats/blob/master/auditbeat/_meta/fields.common.yml#L26

Are you using the new append_fields feature or modifying the original file?

1 Like
3

Nice. Thanks. multi_fields is what I was looking for. I'm modifying the original file. Or better passing a custiom fields.yml via setup.template.fields

Many thanks !

Sirk

4

The feature I was referring to is not really documented yet and is only in master so far. Here you can see the details: https://github.com/elastic/beats/pull/6024

The tricky part before append_fields is that each new version of beats contains new fields (normally) so the fields.yml should be overwritten. But then you have to apply your changes every time.

Can you share a bit more background on what kind of fields you added to fields.yml. This should also help me to better understand how people use it / modify it.

5

Sure. The thing is that we use Filebeat -> ES. No logstash or anything and we log from nginx and go app directly in json to disk.
So Filebeat picks up the already formatted logs from disk and just passed them on to Elasticsearch. This leads to having a totally custom nginx log with our own field-names but more importantly our other apps have custom log format not matching anything out there :wink:
Therefore we need to define our custom mapping.

6

Thanks for the details. JSON logs is definitively a big use case here.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.