Autodiscover with custom pipeline

Elastic Stack Beats
1

Hi

I'm sorry if this has been already asked and explained, but I'm missing it...

My goal is to have a custom grok configuration.

My configuration:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        enabled: false
        type: container
      templates:
        - condition:
            equals:
              kubernetes.namespace: "ingress-nginx"
          config:
            - module: nginx
              access:
                enabled: false
              error:
                enabled: false
              ingress_controller:
                enabled: true
                input:
                  type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}.log
        - condition:
            equals:
                kubernetes.namespace: "my-namespace"
          config:
            - module: nginx
              access:
                enabled: true
                pipeline: /usr/share/filebeat/custom-pipelines/custom-nginx-access-log-pipeline.yml
                ingest_pipeline: /usr/share/filebeat/custom-pipelines/custom-nginx-access-log-pipeline.yml
                input:
                  type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}.log
              error:
                enabled: false
              ingress_controller:
                enabled: false

ingest_pipeline it's what in a module configuration file. But I've read that it's needed pipeline... When it's working I remove the one that it's not needed :wink:
custom-nginx-access-log-pipeline.yml it's a copy of nginx pipeline modified.

I believe that more than one thing is wrong.
The nginx module grok configuration has this %{NUMBER:nginx.ingress_controller.http.request.length:long} after the user_agent.original part. And this is exactly the field I need.
Even so, because it wasn't parsed, I've tried to create my own pipeline, and I finish my expression with %{NUMBER:http.request.time}%{GREEDYDATA:nginx.ingress_controller.http.greed}

In Kibana Grok Debugger everything works. And I already escaped everything correctly. And the pod also have the annotation to enable logging.
But the field never gets parsed.

So, my questions are:
Is it possible to specify my own pipeline like this?
If not, how then?
Is it the rest of the configuration correct?

Thanks in advance

Kind regards,
Emanuel

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.