Require clarification on filebeat ingest pipeline

[Nithani25 (Nishanth Raj)]

Hi Team,
I have installed a brand new setup in my production environment, i am successful in injecting the apache access and error logs via filebeat apache2 module. however, my access log seems to have some additonal data in it like reponse time and user id added to it.

though included those two fields in ingest/default.json i am not able to see them displayed in my kibana. Also my target server has some virtual environment details so i have different environment logs present in the same server,so how can i add new filed referring the environment name and also incldued in the index name.

[kvch (Noémi Ványi)]

Have you updated the pipeline on your Ingest node? Filebeat does not upload pipelines if one exists with the same name. To force refreshing the pipeline you can either set filebeat.override_pipelines to true or run ./filebeat setup -modules=apache2 --pipelines.

In order to index your custom fields, you need to checkout the Beats repository. Add your fields to filebeat/module/apache2/access/_meta/fields.yml and run make update. This updates the index pattern of Filebeat. Run ./filebeat setup -modules=apache2 to update assets on the Ingest node.