# How to filter specific fields of nginx logs in filebeat before importing in elastic?
**URL:** https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475
**Category:** Beats
**Tags:** filebeat
**Created:** [January 20, 2024, 12:54pm UTC](https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475 "2024-01-20T12:54:37Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)
#### Post date: [January 20, 2024, 12:54pm UTC](https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475/1 "2024-01-20T12:54:37Z")
Hi Guys,
I have a filebeat for importing nginx logs to elasticsearch.
this is a sample of my logs:
```auto
5.125. ***** - - [20/Jan/2024:12:50:39 +0000] "GET /findEnemy?_s_=hT%2FF&_u_=7367&baseScore=1831&blevel=11&btid=170536&ctm=1739&eid=13578&platform=android_ir&stype=0&uid=7167 HTTP/1.1" 200 6234 "-" "-" [-]
```
I don't need all of these data. I have some problems with storage and I need a way to filter these logs to import just important data.
I know, with logstash I can do it, but is there anyway to do with filebeat?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 20, 2024, 5:29pm UTC](https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475/2 "2024-01-20T17:29:27Z")
Hi @Siavash_Fazli
Yes with an ingest pipeline you can drop fields
Implementation depends on what version of the stack you are on.
What version of the elastic Stack?
What modules are you using?
---
### Author:  [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)
#### Post date: [January 21, 2024, 6:26am UTC](https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475/3 "2024-01-21T06:26:12Z")
my ELK version: 8.8.2
filebeat version: 8.8.2
I use nginx module with this configuration
```auto
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/main/filebeat-module-nginx.html
- module: nginx
# Access logs
access:
enabled: true
var.paths: ["/data/logs/nginx/access.log*", "/data/logs/nginx/postdata-access.log*"]
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Error logs
error:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
ingress_controller:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
```
and this is my filebeat.yml configuration
```auto
# ==== Filebeat inputs =======
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: false
paths:
- /var/log/*.log
# ==== Filebeat modules ===
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 30s
# ==== Elasticsearch template setting ====
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
setup.dashboards.index: "filebeat-*"
#setup.template.enabled: false
output.elasticsearch.index: "filebeat-coz-%{[agent.version]}"
setup.template.name: "filebeat-coz-%{[agent.version]}"
setup.template.pattern: "filebeat-coz-%{[agent.version]}"
setup.template.overwrite: false
setup.template.fields: "/etc/filebeat/fields-coz.yml"
setup.ilm.enabled: true
setup.ilm.overwrite: true
setup.ilm.policy_name: "filebeat-coz-daily"
setup.ilm.check_exists: true
setup.ilm.policy_file: "/etc/filebeat/ilm.json"
# ================ Kibana ===========
setup.kibana:
host: "https:// ********* om:443"
# ------------- Elasticsearch Output ----------
output.elasticsearch:
hosts: ["https:// ******** com:443"]
protocol: "https"
api_key: " ************************"
# == Processors =======
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 21, 2024, 4:18pm UTC](https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475/4 "2024-01-21T16:18:24Z")
Hi @Siavash_Fazli
So there are 2 ways to do this
1. Easiest but perhaps not best long term : Add a `remove` processor to the end of the existing pipeline
2. Clone the pipeline and then add the `remove` processor to your customer pipeline and use that pipeline in the module [Here](https://discuss.elastic.co/t/help-parsing-custom-nginx-logs-using-filebeat-and-ingest-pipelines/349974/6) are detailed instructions for that ... but for you instead of all the `grok` stuff you would just add the `remove` processor as the last processor..
So for Number 1)
Go to Kibana - Stack Management - Ingest Pipeline
Find the nginx access pipeline - 1st Clone it for a backup, then Edit the original pipeline
at the bottom of the normal processors add a `remove` processor and list the fields you want to remove... IMPORTANT don't forget to save the processor and pipeline
Let us know how it works...
---
### Author:  [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)
#### Post date: [January 24, 2024, 8:21am UTC](https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475/5 "2024-01-24T08:21:00Z")
Thank you. I'll apply your recommendations.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 21, 2024, 10:21am UTC](https://discuss.elastic.co/t/how-to-filter-specific-fields-of-nginx-logs-in-filebeat-before-importing-in-elastic/351475/6 "2024-02-21T10:21:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.