# Help parsing custom nginx logs using Filebeat and Ingest Pipelines

**URL:** <https://discuss.elastic.co/t/help-parsing-custom-nginx-logs-using-filebeat-and-ingest-pipelines/349974>\
**Category:** Elasticsearch\
**Created:** [December 26, 2023, 4:39pm UTC](https://discuss.elastic.co/t/help-parsing-custom-nginx-logs-using-filebeat-and-ingest-pipelines/349974 "2023-12-26T16:39:24Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 28, 2023, 2:19am UTC](https://discuss.elastic.co/t/help-parsing-custom-nginx-logs-using-filebeat-and-ingest-pipelines/349974/6 "2023-12-28T02:19:52Z")

</div>

Hi @BDeveloper

I have and update / solution for you. You may need to adjust etc... I picked some names for fields etc, there is no guarantee that every log line will parse / work.

We will start from where you left off with the Quickstart setting up the Filebeat and nginx module.

Then The Macro Steps are

- Clone the Existing nginx Access Module Pipeline to become our custom pipeline
- Add the new Grok Pattern Etc to support your custom format
- Set the the modules to use our new custom pipeline
- Run filebeat

Here are the files

[The Pipeline Custom Pipeline](https://gist.github.com/bvader/6341b1dc5db3a2b0a1728ce931b3191f)

[The Log File](https://gist.github.com/bvader/28c550b919cd509c12285fe6892d45e5)

[The nginx.yml](https://gist.github.com/bvader/421ed0e907897f0bb9a57afaec00b3dc)

In the next Post (perhaps tomorrow) I will show you how to quickly build / test ingest pipeline / groks etc.

These are the steps... .follow them very very closely. I am doing them through the UI but of course in reality I do it all through the API. with the links I put above

Clone The Existing Ingest Pipeline, Name It

 ![Screenshot 2023-12-27 at 5.38.47 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/f/3f31141e279400e38f95699bcef57418a0ee9510.png)

 ![Screenshot 2023-12-27 at 5.56.35 PM](https://us1.discourse-cdn.com/elastic/original/3X/2/4/243c909679bd26473eb0abb38af91b4e57024dd6.jpeg)

Add the new Grok Pattern and  
**IMPORTANT** Move it to the top so that it matches first (perhaps more on that later) Be Careful with the cut and past.  
Save the Grok

`(%{NGINX_HOST} )?"?(?:%{NGINX_ADDRESS_LIST:nginx.access.remote_ip_list}|%{NOTSPACE:source.address}) - (-|%{DATA:user.name}) \[%{HTTPDATE:nginx.access.time}\] "%{DATA:nginx.access.info}" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} "(-|%{DATA:http.request.referrer})" "(-|%{DATA:user_agent.original})" "-" "(-|%{IPORHOST:nginx.access.host.name})" sn="(-|%{DATA:nginx.access.host.domain})" rt=(-|%{NUMBER:nginx.access.request_time:float}) ua="(-|%{DATA:nginx.access.upstream_addr})" us="(-|%{DATA:nginx.access.upstream_status})" ut="(-|%{NUMBER:nginx.access.upstream_response_time:float})" ul="(-|%{NUMBER:nginx.access.upstream_response_length:long})" cs=-`

 ![Screenshot 2023-12-27 at 5.57.22 PM](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0e67a93acb7015884dae5fd6473dd4643a5a90b0.png)

**IMPORTANT** Save the Pipeline

 ![Screenshot 2023-12-27 at 5.39.25 PM](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5e4c1128bcec44245abebd7b1ee712dccceb1dbc.png)

Now Modify the `nginx.yml` to use the new custom pipeline.

```auto
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set the custom pipeline
    input.pipeline: filebeat-8.11.3-nginx-access-pipeline-custom

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/Users/sbrown/workspace/sample-data/discuss/discuss-custom-nginx.log"]

```

Start filebeat

Check Discover

 ![Screenshot 2023-12-27 at 6.10.21 PM](https://us1.discourse-cdn.com/elastic/original/3X/f/5/f57cb4ed9e862b2b5f8c34e25f7b10f67195f679.jpeg)

 ![Screenshot 2023-12-27 at 6.10.45 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/7/d785d7e26cf5ccbcf5326eb14f060eca90adc981.jpeg)

---

_[View the full topic](https://discuss.elastic.co/t/help-parsing-custom-nginx-logs-using-filebeat-and-ingest-pipelines/349974)._