Filebeat Field Mapping For JSON Logs

I have created a Spring application which outputs logfiles as JSON after attending an Elasticsearch meetup which is based on.

However when Filebeats creates the index template under the json section it defines one of the mapping fields as a keyword. This field should be of type text so that I can do a full text search as it is unstructured text

I found this issue in the forum but I'm not sure of what the outcome was

It seems to suggest I can use setup.template.append_fields to configure how the fields in my JSON logs should be mapped

There doesn't seem to be clear documentation on how I can map the JSON logs. There appears to be an option between

  • Exporting a JSON template from filebeat and uploading it to Elasticsearch
  • Creating a fields.yml
  • Using setup.template.append_fields

I only want to configure a single field in the JSON to be text, the rest can remain as keywords. Is it possible to only configure the fields I want and leave the others as defaults. I'm happy with the default mappings that filebeat creates for each of the modules, for example I am using docker autodiscovery and want to use the docker module for the docker metadata. I would like to keep the maintenance of the mappings low so I only configure changes to the defaults.

This issue which was resolved by

Is there good documentation on configuring the mappings for fields in the JSON logs?

I think that if you add this to your template then it will add a mapping specifically for the field to make it text rather than keyword.

- name:
  type: text

You can verify the generated template with:

filebeat export template

Note that after changing the template you will need force Filebeat to write the updated index template to Elasticsearch (I use filebeat setup --template -E setup.template.overwrite=true). The new index template will only apply to newly created indices and not existing ones.

And I suggest opening a new Github issue for the documentation around setup.template.append_fields. Having an example would be useful.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.