How to filter specific fields of nginx logs in filebeat before importing in elastic?

[Siavash_Fazli]

Hi Guys,
I have a filebeat for importing nginx logs to elasticsearch.
this is a sample of my logs:

5.125.***** - - [20/Jan/2024:12:50:39 +0000] "GET /findEnemy?_s_=hT%2FF&_u_=7367&baseScore=1831&blevel=11&btid=170536&ctm=1739&eid=13578&platform=android_ir&stype=0&uid=7167 HTTP/1.1" 200 6234 "-" "-" [-]

I don't need all of these data. I have some problems with storage and I need a way to filter these logs to import just important data.
I know, with logstash I can do it, but is there anyway to do with filebeat?

[stephenb (Stephen Brown)]

Hi @Siavash_Fazli

Yes with an ingest pipeline you can drop fields

Implementation depends on what version of the stack you are on.

What version of the elastic Stack?

What modules are you using?

[Siavash_Fazli]

my ELK version: 8.8.2
filebeat version: 8.8.2
I use nginx module with this configuration

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/main/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["/data/logs/nginx/access.log*", "/data/logs/nginx/postdata-access.log*"]
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
  # Error logs
  error:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

and this is my filebeat.yml configuration

#  ==== Filebeat inputs =======

filebeat.inputs:

- type: filestream
  id: my-filestream-id
  enabled: false
  paths:
    - /var/log/*.log
 
# ==== Filebeat modules ===

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s

# ==== Elasticsearch template setting ====

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
setup.dashboards.index: "filebeat-*"
#setup.template.enabled: false
output.elasticsearch.index: "filebeat-coz-%{[agent.version]}"
setup.template.name: "filebeat-coz-%{[agent.version]}"
setup.template.pattern: "filebeat-coz-%{[agent.version]}"
setup.template.overwrite: false
setup.template.fields: "/etc/filebeat/fields-coz.yml"
setup.ilm.enabled: true
setup.ilm.overwrite: true
setup.ilm.policy_name: "filebeat-coz-daily"
setup.ilm.check_exists: true
setup.ilm.policy_file: "/etc/filebeat/ilm.json"

# ================ Kibana ===========

setup.kibana:

  host: "https://*********om:443"

# ------------- Elasticsearch Output ----------
output.elasticsearch:
  hosts: ["https://********com:443"]
  protocol: "https"

  api_key: "************************"


# == Processors =======
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

[stephenb (Stephen Brown)]

Hi @Siavash_Fazli

So there are 2 ways to do this

1. Easiest but perhaps not best long term : Add a remove processor to the end of the existing pipeline

2. Clone the pipeline and then add the remove processor to your customer pipeline and use that pipeline in the module Here are detailed instructions for that ... but for you instead of all the grok stuff you would just add the remove processor as the last processor..

So for Number 1)

Go to Kibana - Stack Management - Ingest Pipeline

Find the nginx access pipeline - 1st Clone it for a backup, then Edit the original pipeline

at the bottom of the normal processors add a remove processor and list the fields you want to remove... IMPORTANT don't forget to save the processor and pipeline

Let us know how it works...

[Siavash_Fazli]

Thank you. I'll apply your recommendations.