Custom fields.yml

Elastic Stack Beats
1

Hi,

  • I have defined template in ES via _template api.
  • Created index also , having mapping as per above template.
    Now i want to feed data hence i have setup filebeat -> Logstash -> ES.
    I have created dummy data as per index mapping and pass as a source to filebeat.

End to end data flow happening but with below issue:-

I am getting my dummy data in "message":"<>" property of filebeat template and updating index mapping also . My question is how to handle this issue,i dont want the filebeat fields in this index.

  1. Write a gork in Logstash to get the dummy data from "message" property?
  2. Can we pass the same(defined in ES via api) template to file beat and not use fields.yml?
  3. Is it possible to convert template.json into *.yml via some utility?

Please suggest.
Regards
Ajay

2

fields.yml files are the input to the index mapping generator of Beats. If you already deployed your own index and do not want any additional fields from Beats, just don't set setup.template.enabled to false. Make sure to set the index option of your configured input, so Filebeat forwards it to the appropriate index.

3

hi,
after setting setup.template.enabled: false , stilling getting filebeat fields in document.

I am sending logs like filebeat -> logstash -> ES so i think index info i have provided in logstash.

After enabling json.keys_under_root: true and one json object in one line in log, i can see the dummy data in document but some extra filebeat fields are still coming.