# Unexpected behavior for autodiscover appenders with processors

**URL:** <https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 8, 2019, 3:34am UTC](https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415 "2019-03-08T03:34:12Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![walfie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/walfie/32/41691_2.png) [@walfie](https://discuss.elastic.co/u/walfie)\
**Post date:** [March 8, 2019, 3:34am UTC](https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415/1 "2019-03-08T03:34:12Z")

</div>

I'm trying to use autodiscover, where I have some processors defined in the `templates` config, as well as some processors defined in the `appenders` section under certain conditions, like so:

```auto
filebeat.autodiscover:
  providers:
  - type: kubernetes
    templates:
    - config:
      - type: docker
        containers.ids:
        - "${data.kubernetes.container.id}"
        fields_under_root: true

        processors:
        - rename:
            fields:
            - from: kubernetes.container
              to: blah
            ignore_missing: true
            fail_on_error: false

    appenders:
    - type: config
      condition.equals.kubernetes.labels.k8s-app: filebeat
      config:
        processors:
        - decode_json_fields:
            fields: ["message"]
            target: ""
            overwrite_keys: true
        - drop_fields:
            fields: ["timestamp"]

```

What I would expect to happen is for the two `processors` arrays to be concatenated.

However, I keep getting the error "Error creating runner from config: each processor needs to have exactly one action, but found 2 actions" which confused me, because the processors looked correct to me.

I think I was able to figure out the issue though, it's that when filebeat attempts to combine the `appenders` config with the `templates` config, it may be flattening the array representations and combining each of the individual items with the same array index.

i.e., I imagine it was attempting to combine my `rename` processor with my `decode_json_fields` processor into one object, like:

```auto
- rename:
    fields:
    - from: kubernetes.container
      to: blah
    ignore_missing: true
    fail_on_error: false
  decode_json_fields: # note this is on the same level as `rename`
    fields: ["message"]
    target: ""
    overwrite_keys: true

```

which is very much an invalid processor.

I was able to work around it by changing my `appenders` definition to use numbered syntax for arrays, with numbers that don't overlap with the ones defined above:

```auto
    appenders:
    - type: config
      condition.equals.kubernetes.labels.k8s-app: filebeat
      config:
        processors.1.decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true
        processors.2.drop_fields:
          fields: ["timestamp"]

```

Now clearly this isn't an optimal solution, as I would expect the appender to just concatenate the arrays. Has anyone else encountered this issue? I'm encountering it on 7.0.0-beta1 but I imagine it's present in 6.x as well.

---

<div class="post-metadata">

**Author:** ![walfie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/walfie/32/41691_2.png) [@walfie](https://discuss.elastic.co/u/walfie)\
**Post date:** [March 8, 2019, 11:40pm UTC](https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415/2 "2019-03-08T23:40:34Z")

</div>

Here's a specific location in the code where this can be tested: [https://github.com/elastic/beats/blob/7.x/libbeat/autodiscover/appenders/config/config\_test.go](https://github.com/elastic/beats/blob/7.x/libbeat/autodiscover/appenders/config/config_test.go)

I was able to confirm it locally by adding a test that had an array field containing objects in both the `eventConfig` and `config` fields, and was seeing that the objects at each array index were being merged.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [March 11, 2019, 2:55pm UTC](https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415/3 "2019-03-11T14:55:06Z")

</div>

I think your workaround is the only way how to solve it currently.  
Can you open an bug report on github please: [https://github.com/elastic/beats/issues/new/choose](https://github.com/elastic/beats/issues/new/choose)

---

<div class="post-metadata">

**Author:** ![walfie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/walfie/32/41691_2.png) [@walfie](https://discuss.elastic.co/u/walfie)\
**Post date:** [March 11, 2019, 3:13pm UTC](https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415/4 "2019-03-11T15:13:12Z")

</div>

Sure, I've opened an issue here: [https://github.com/elastic/beats/issues/11184](https://github.com/elastic/beats/issues/11184)

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [March 11, 2019, 3:25pm UTC](https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415/5 "2019-03-11T15:25:53Z")

</div>

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 8, 2019, 3:26pm UTC](https://discuss.elastic.co/t/unexpected-behavior-for-autodiscover-appenders-with-processors/171415/6 "2019-04-08T15:26:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.