# Autodiscover with custom pipeline

**URL:** https://discuss.elastic.co/t/autodiscover-with-custom-pipeline/318377
**Category:** Beats
**Tags:** filebeat
**Created:** [November 8, 2022, 2:00am UTC](https://discuss.elastic.co/t/autodiscover-with-custom-pipeline/318377 "2022-11-08T02:00:32Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![zeitler](https://avatars.discourse-cdn.com/v4/letter/z/4491bb/32.png) [@zeitler](https://discuss.elastic.co/u/zeitler)
#### Post date: [November 8, 2022, 2:00am UTC](https://discuss.elastic.co/t/autodiscover-with-custom-pipeline/318377/1 "2022-11-08T02:00:32Z")

</div>

Hi

I'm sorry if this has been already asked and explained, but I'm missing it...

My goal is to have a custom grok configuration.

My configuration:

```auto
filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        enabled: false
        type: container
      templates:
        - condition:
            equals:
              kubernetes.namespace: "ingress-nginx"
          config:
            - module: nginx
              access:
                enabled: false
              error:
                enabled: false
              ingress_controller:
                enabled: true
                input:
                  type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}.log
        - condition:
            equals:
                kubernetes.namespace: "my-namespace"
          config:
            - module: nginx
              access:
                enabled: true
                pipeline: /usr/share/filebeat/custom-pipelines/custom-nginx-access-log-pipeline.yml
                ingest_pipeline: /usr/share/filebeat/custom-pipelines/custom-nginx-access-log-pipeline.yml
                input:
                  type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}.log
              error:
                enabled: false
              ingress_controller:
                enabled: false

```

ingest\_pipeline it's what in a module configuration file. But I've read that it's needed pipeline... When it's working I remove the one that it's not needed 😉  
custom-nginx-access-log-pipeline.yml it's a copy of nginx pipeline modified.

I believe that more than one thing is wrong.  
The nginx module grok configuration has this `%{NUMBER:nginx.ingress_controller.http.request.length:long}` after the user\_agent.original part. And this is exactly the field I need.  
Even so, because it wasn't parsed, I've tried to create my own pipeline, and I finish my expression with ` %{NUMBER:http.request.time}%{GREEDYDATA:nginx.ingress_controller.http.greed}`

In Kibana Grok Debugger everything works. And I already escaped everything correctly. And the pod also have the annotation to enable logging.  
But the field never gets parsed.

So, my questions are:  
Is it possible to specify my own pipeline like this?  
If not, how then?  
Is it the rest of the configuration correct?

Thanks in advance

Kind regards,  
Emanuel

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 6, 2022, 4:01am UTC](https://discuss.elastic.co/t/autodiscover-with-custom-pipeline/318377/2 "2022-12-06T04:01:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.