# Using custom pipeline for existing filebeat module

**URL:** <https://discuss.elastic.co/t/using-custom-pipeline-for-existing-filebeat-module/213270>\
**Category:** Beats\
**Tags:** beats-module, filebeat\
**Created:** [December 28, 2019, 11:02pm UTC](https://discuss.elastic.co/t/using-custom-pipeline-for-existing-filebeat-module/213270 "2019-12-28T23:02:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jsosic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsosic/32/5945_2.png) [@jsosic](https://discuss.elastic.co/u/jsosic)\
**Post date:** [December 28, 2019, 11:02pm UTC](https://discuss.elastic.co/t/using-custom-pipeline-for-existing-filebeat-module/213270/1 "2019-12-28T23:02:21Z")

</div>

Hi guys.

I use custom nginx log format, with some additional fields, so current pipeline in Filebeat 7.x nginx module fails with "grok parse failure".

My platform is `CentOS 7.x`.

Now, I'm thinking of what is the best way to reuse the module and only change the pipeline parser line?

**Option 1:**  
Overwrite `default.json` with my custom one.

- Pros: quick and easy
- Cons: have to be re-done after each upgrade, so simple `yum update` doesn't suffice any more, plus it's not a good practice overwriting files in `/usr`.

**Option 2:**  
Copying whole nginx module to `mycom_nginx` and changing `default.json` there, enabling `mycom_nginx` and disabling `nginx` module.

- Pros: quick and easy
- Cons: have to keep the module up to date whenever it's changed upstream, plus have to understand all the details of the module like machine learning part.

I don't like either of these two... but I can't figure out how to still use the `nginx` module, but just specify different pipeline file in `/etc/filebeat/modules.d/nginx.yml`.

How do you guys do it?

---

<div class="post-metadata">

**Author:** ![jsosic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsosic/32/5945_2.png) [@jsosic](https://discuss.elastic.co/u/jsosic)\
**Post date:** [January 21, 2020, 12:31am UTC](https://discuss.elastic.co/t/using-custom-pipeline-for-existing-filebeat-module/213270/2 "2020-01-21T00:31:50Z")

</div>

I ended up doing the following:

deploying custom pipelines via `module/nginx/error/ingest/custom.json` and modified `nginx/error/manifest.yml` to look like:

```auto
module_version: "1.0"

var:
  - name: paths
    default:
      - /var/log/nginx/error.log*
    os.darwin:
      - /usr/local/var/log/nginx/error.log*
    os.windows:
      - c:/programdata/nginx/logs/error.log*

ingest_pipeline: ingest/custom.json
input: config/nginx-error.yml

```

Only line changed is `ingest_pipeline`.

After this, I run:

```auto
filebeat setup --pipelines

```

and that's it.

If a better method is discovered, I'll modify my approach.

---

<div class="post-metadata">

**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\
**Post date:** [January 23, 2020, 6:50pm UTC](https://discuss.elastic.co/t/using-custom-pipeline-for-existing-filebeat-module/213270/3 "2020-01-23T18:50:39Z")

</div>

Hi @jsosic 🙂

The problem is that you are not using Nginx module at the end so, any solution will involve maintaining code. Your first option maybe is less prone to errors. The key thing here is that the pipeline is a JSON file where you can update the array with a new Grok pattern writing a simple script that you can run every time you update.

You can also try to use some processors in the input part (before the module) if you feel like you can "extract" the non standard data from the incoming line before it reaches the processor. [https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html) Those processors are Filebeat processors, do not confuse them with Ingest Node processors 😅

I hope this helps

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 20, 2020, 6:50pm UTC](https://discuss.elastic.co/t/using-custom-pipeline-for-existing-filebeat-module/213270/4 "2020-02-20T18:50:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.