Gradual migration from container input to kubernetes autodiscover

[OranShuster]

We are currently using a filebeat daemon set with the "old" container input, both version 7.17.x
we want to start migrating to the newer approach of hint based log collection
for this we need the old filebeat daemon set to run alongside the new autodiscover deployment
for hint based it's easy to only collect logs from pods that use the annotations since we can turn the default configuration off
The problem is we now want the old filebeat to filter out any pod/container that is handled by the auto discover
Since add_kubernetes_metadata does not add pod annotations we cannot use drop_event with a condition based on annotations

Other than manually adding a label in addition to the annotations, is there a solution for running the 2 types of inputs at the same time?

[Michalis_Katsoulis]

Hi @OranShuster ,

I can see that this is not included in the available configuration options of add_kubernetes_metadata processor ( Add Kubernetes metadata | Filebeat Reference [7.17] | Elastic) but you can add pod annotations.

The way to do it is to use the include_annotations setting.
Example:

  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            include_annotations: ["app"]
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

[OranShuster]

When you say "not included in the available configuration options..." you mean it's undocumented?
from the docs i see we can add resource annotations (nodes,namespaces) but not for pods.
Ill try your suggestion anyway, currently we decided to add a label and filter by that

[Michalis_Katsoulis]

Yes it is undocumented. Keep in mind that this option is not part of the add_resource_metadata. There you can specify which metadata of nodes and namespaces you want the events to be enriched with.
It has to be added on the higher level of the processor configuration.