Autodiscover in Beats doesn't seem to see my container when I update it

Elastic Stack Beats
,
1

I have finally been able to configure Beats in the way that I need. That is to capture logs only from selected containers. I'm using processors.drop_event to exclude the containers that I don't want.
I'm also using autodiscover. As far as I understand, autodiscover should pick-up any container that matches my criteria. I.e. if I start a new container that matches the condition, Beats should automatically pick it up.
Well, that part doesn't work.
Here is my full yaml:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: jsc-filebeat
  namespace: jsc-ns
spec:
  type: filebeat
  version: 8.17.2
  config:
    filebeat:
      inputs:
        - type: container
          paths:
            - /var/log/containers/*.log
      autodiscover:
        providers:
          - type: kubernetes
            node: ${NODE_NAME}
            templates:
              - config:
                  - type: container
                    paths:
                      - /var/log/containers/*${data.kubernetes.container.id}.log
                conditions:
                  or:
                    - contains.kubernetes.container.name: wordpress-app
                    - contains.kubernetes.container.name: my-other-app
            hints:
              enabled: true
    output:
      logstash:
        hosts: ["jsc-logstash-ls-beats:5044"]
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}
      - drop_event:
          when:
            not:
              or:
                - contains.kubernetes.container.name: wordpress-app
                - contains.kubernetes.container.name: my-other-app
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        containers:
          - name: filebeat
            securityContext:
              runAsUser: 0
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers

I am expecting that if I deploy an update to container with the name my-other-app, Beats should pick-up the logs automatically. However it doesn't.
I see my 2 containers logs in kibana, then as soon as I apply a change to the my-other-app container, it stops getting picked by Beats and no more data for this container in Elasticsearch

Is there something more I need to add or remove in my configuration?

2

Hmmm quick eyeball

                      - /var/log/containers/*${data.kubernetes.container.id}.log
                conditions: <<< No s
                  or:
...

Perhaps per docs Should be

condition