Im trying to create my own filebeat. I want to work with the direct elasticbeat ingest Pipeline (no Logstash in between).
I downloaded the filebeat development package from github and programmed my filebeat module. I made the GROK for the ingest pipeline tested it with the "_ingest/pipeline/_simulate" and with the make command i got the "mymodule/access/_meta/fields.yml".
When i copy this module now to an existing filebeat installation (copy the module folder to /usr/share/filebeat/modules/", newly created index indexes the data, but dont see the fields as i have defined them in my _meta/fields.yml.
The indexed document adds all fields as "keyword" and if i look at the "_template/filebeat-7.0.0" my definitions are also not there (i assume thats why my fields all get indexed as keyword).
Also if i delete all indicies from elasticseach and the "_template/filebeat-7.0.0" my fields dont show up after automatic recreation.
As it seems the Global "/etc/filebeat/fields.yml" is the only structure that is transferred to elasticseach upon first indexing from a filebeat.
Is there any way to add fields from my module to the index definition without modifying the Global "/etc/filebeat/fields.yml" or is there another way to acomplish this?
TIA Rolf