How to filter specific fields of nginx logs in filebeat before importing in elastic?

[stephenb (Stephen Brown)]

Hi @Siavash_Fazli

So there are 2 ways to do this

1. Easiest but perhaps not best long term : Add a remove processor to the end of the existing pipeline

2. Clone the pipeline and then add the remove processor to your customer pipeline and use that pipeline in the module Here are detailed instructions for that ... but for you instead of all the grok stuff you would just add the remove processor as the last processor..

So for Number 1)

Go to Kibana - Stack Management - Ingest Pipeline

Find the nginx access pipeline - 1st Clone it for a backup, then Edit the original pipeline

at the bottom of the normal processors add a remove processor and list the fields you want to remove... IMPORTANT don't forget to save the processor and pipeline

Let us know how it works...