Scenario
- A shared K8s cluster with several different apps all needing their app-specific logs collected by Filebeat.
- Each different app's logs are substantially different (some plaintext, some json with differing fields)
- Each different app's logs have varying volumes (some generate a little, some a lot)
- We want each app to have their own set of indices with their own ILM policies and possibly different sharding as volume dictates.
- Each app will have its own credentials to protect private indices from accidental writes from other apps.
Question
How can we utilize Filebeat on K8s/EKS as described here using the Autodiscovery feature?
When the Autodiscover feature's templates don't allow for specifying the output
Filebeat supports templates for inputs and modules
And Filebeat itself only allows a single output:
Only a single output may be defined.
And the target index is specified in the output
There is a indices setting on the output that allows for conditionals but it looks like all conditionals are based upon the message/event fields and not anything about K8s fields. Also we cannot see how to supply different credentials to the output.
In other words it looks like our options are limited. The indices way like this:
output.elasticsearch:
hosts: ["http://localhost:9200"]
# This will get unwieldy as more and more apps run on the cluster
indices:
- index: "app-private-ilm-alias"
when.contains:
# We don't care what a field in the event says (which also wont work well for plaintext logs), we want to target an index by an application's id from K8s
message: "WARN"
The autodiscover way like this:
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
# This could also get unwieldy as the number of applications increases
templates:
- condition:
contains:
# This would target the application we want, but we cannot specify the output index this way
kubernetes.container.name: "no-json-logging"
config:
- type: container
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
What are the other options? Is there a way to target different indices and use autodiscover while also having separate credentials? Or would we need to go through the trouble of involving a Logstash layer as an intermediary + bulkhead between Filebeat and Elasticsearch?