Filebeat Field Mapping For JSON Logs

Elastic Stack Beats
1

I have created a Spring application which outputs logfiles as JSON after attending an Elasticsearch meetup which is based on.

https://github.com/xeraa/microservice-monitoring

However when Filebeats creates the index template under the json section it defines one of the mapping fields json.rest as a keyword. This field should be of type text so that I can do a full text search as it is unstructured text

I found this issue in the forum but I'm not sure of what the outcome was

https://discuss.elastic.co/t/custom-filebeat-template-for-json-log-lines/114761

It seems to suggest I can use setup.template.append_fields to configure how the fields in my JSON logs should be mapped https://www.elastic.co/guide/en/beats/filebeat/6.3/configuration-template.html

There doesn't seem to be clear documentation on how I can map the JSON logs. There appears to be an option between

  • Exporting a JSON template from filebeat and uploading it to Elasticsearch
  • Creating a fields.yml
  • Using setup.template.append_fields

I only want to configure a single field in the JSON to be text, the rest can remain as keywords. Is it possible to only configure the fields I want and leave the others as defaults. I'm happy with the default mappings that filebeat creates for each of the modules, for example I am using docker autodiscovery and want to use the docker module for the docker metadata. I would like to keep the maintenance of the mappings low so I only configure changes to the defaults.

This issue
https://github.com/elastic/beats/issues/1427 which was resolved by https://github.com/elastic/beats/pull/1472

Is there good documentation on configuring the mappings for fields in the JSON logs?

2

I think that if you add this to your template then it will add a mapping specifically for the json.rest field to make it text rather than keyword.

setup.template.append_fields:
- name: json.rest
  type: text

You can verify the generated template with:

filebeat export template

Note that after changing the template you will need force Filebeat to write the updated index template to Elasticsearch (I use filebeat setup --template -E setup.template.overwrite=true). The new index template will only apply to newly created indices and not existing ones.

3

And I suggest opening a new Github issue for the documentation around setup.template.append_fields. Having an example would be useful.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.