Filebeat autodiscovery in Kubernetes/EKS and multiple outputs

Scenario

  • A shared K8s cluster with several different apps all needing their app-specific logs collected by Filebeat.
  • Each different app's logs are substantially different (some plaintext, some json with differing fields)
  • Each different app's logs have varying volumes (some generate a little, some a lot)
  • We want each app to have their own set of indices with their own ILM policies and possibly different sharding as volume dictates.
  • Each app will have its own credentials to protect private indices from accidental writes from other apps.

Question
How can we utilize Filebeat on K8s/EKS as described here using the Autodiscovery feature?

When the Autodiscover feature's templates don't allow for specifying the output

Filebeat supports templates for inputs and modules

And Filebeat itself only allows a single output:

Only a single output may be defined.

And the target index is specified in the output

There is a indices setting on the output that allows for conditionals but it looks like all conditionals are based upon the message/event fields and not anything about K8s fields. Also we cannot see how to supply different credentials to the output.

In other words it looks like our options are limited. The indices way like this:

    output.elasticsearch:
      hosts: ["http://localhost:9200"]
      # This will get unwieldy as more and more apps run on the cluster
      indices:
        - index: "app-private-ilm-alias"
          when.contains:
           # We don't care what a field in the event says (which also wont work well for plaintext logs), we want to target an index by an application's id from K8s
            message: "WARN"

The autodiscover way like this:

    filebeat.autodiscover:
      providers:
          - type: kubernetes
            node: ${NODE_NAME}
            # This could also get unwieldy as the number of applications increases
            templates:
              - condition:
                  contains:
                    # This would target the application we want, but we cannot specify the output index this way
                    kubernetes.container.name: "no-json-logging"
                config:
                  - type: container
                    paths:
                      - "/var/log/containers/*-${data.kubernetes.container.id}.log"

What are the other options? Is there a way to target different indices and use autodiscover while also having separate credentials? Or would we need to go through the trouble of involving a Logstash layer as an intermediary + bulkhead between Filebeat and Elasticsearch?

Opening a Support case. We need guidance.