How to configure processor in autodiscover mode?

Elastic Stack Beats
,
1

Hi,
We have k8s cluster with filebeat configured in autodiscover mode with hints:

filebeat.yml: |
  logging.level: debug
  filebeat.autodiscover:
    providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        hints.default_config:
          enabled: false
          type: docker
          containers.ids:
            - ${data.kubernetes.container.id}
  output.elasticsearch:
    host: '${NODE_NAME}'
    hosts: 'x.x.x.x:xxxx'

To enable nginx module the following annotations were added to appropriate pods:

  podAnnotations:
    co.elastic.logs/enabled: "true"
    co.elastic.logs/fileset.stderr: error
    co.elastic.logs/fileset.stdout: ingress_controller
    co.elastic.logs/module: nginx

It has been working great and convenient until we needed to convert some fields from nginx. I've tried to reach that by something like that:

filebeat.yml: |
  logging.level: debug
  filebeat.autodiscover:
    providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        hints.default_config:
          enabled: false
          type: docker
          containers.ids:
            - ${data.kubernetes.container.id}
  processors:
    - add_labels:
        labels:
          test_label: tested
    - copy_fields:
        fields:
          - from: "nginx.ingress_controller.upstream.ip"
            to: "nginx.ingress_controller.upstream.ip_test"
        fail_on_error: false
        ignore_missing: true
    - script:
        lang: javascript
        source: >
          function process(event) {
            var value = event.Get("nginx.ingress_controller.upstream.ip")
            event.Put("my_test_value", value);
            }
  output.elasticsearch:
    host: '${NODE_NAME}'
    hosts: 'x.x.x.x:xxxx'

Unfortunately the only labels.test_label - field was added as expected, nginx.ingress_controller.upstream.ip_test - doesn't appear at all and my_test_value is empty. Looks like processors run before nginx module does it job and have no access to appropriate fields. Is it any way to process nginx module fields without using logstash (we are not using it at the moment) and Ingest Node Pipelines?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.