Unexpected behavior for autodiscover appenders with processors

[walfie]

I'm trying to use autodiscover, where I have some processors defined in the templates config, as well as some processors defined in the appenders section under certain conditions, like so:

filebeat.autodiscover:
  providers:
  - type: kubernetes
    templates:
    - config:
      - type: docker
        containers.ids:
        - "${data.kubernetes.container.id}"
        fields_under_root: true

        processors:
        - rename:
            fields:
            - from: kubernetes.container
              to: blah
            ignore_missing: true
            fail_on_error: false

    appenders:
    - type: config
      condition.equals.kubernetes.labels.k8s-app: filebeat
      config:
        processors:
        - decode_json_fields:
            fields: ["message"]
            target: ""
            overwrite_keys: true
        - drop_fields:
            fields: ["timestamp"]

What I would expect to happen is for the two processors arrays to be concatenated.

However, I keep getting the error "Error creating runner from config: each processor needs to have exactly one action, but found 2 actions" which confused me, because the processors looked correct to me.

I think I was able to figure out the issue though, it's that when filebeat attempts to combine the appenders config with the templates config, it may be flattening the array representations and combining each of the individual items with the same array index.

i.e., I imagine it was attempting to combine my rename processor with my decode_json_fields processor into one object, like:

- rename:
    fields:
    - from: kubernetes.container
      to: blah
    ignore_missing: true
    fail_on_error: false
  decode_json_fields: # note this is on the same level as `rename`
    fields: ["message"]
    target: ""
    overwrite_keys: true

which is very much an invalid processor.

I was able to work around it by changing my appenders definition to use numbered syntax for arrays, with numbers that don't overlap with the ones defined above:

    appenders:
    - type: config
      condition.equals.kubernetes.labels.k8s-app: filebeat
      config:
        processors.1.decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true
        processors.2.drop_fields:
          fields: ["timestamp"]

Now clearly this isn't an optimal solution, as I would expect the appender to just concatenate the arrays. Has anyone else encountered this issue? I'm encountering it on 7.0.0-beta1 but I imagine it's present in 6.x as well.

[walfie]

Here's a specific location in the code where this can be tested: https://github.com/elastic/beats/blob/7.x/libbeat/autodiscover/appenders/config/config_test.go

I was able to confirm it locally by adding a test that had an array field containing objects in both the eventConfig and config fields, and was seeing that the objects at each array index were being merged.

[steffens (Steffen Siering)]

I think your workaround is the only way how to solve it currently.
Can you open an bug report on github please: https://github.com/elastic/beats/issues/new/choose

[walfie]

Sure, I've opened an issue here: https://github.com/elastic/beats/issues/11184

[steffens (Steffen Siering)]

Thank you