Filebeat [7.15.0] hints based discovery does nothing

Continuing the discussion from Filebeat doesn't send logs to Elasticsearch:

We're having the exact same problem as the above which doesn't appear to have ever been resolved.
Using filebeat 7.15.0, exact same config as above, running on kind, we have a very simple test setup.

  • filebeat configured as above to write to logstash
  • logstash configured to write to stdout
  • podinfo installed

To reproduce:

  1. Start kind (or whatever k8s cluster you want to use)
  2. Install podinfo using helm3 helm upgrade --install --wait backend -f podinfo-values.yaml --set redis.enabled=true podinfo/podinfo where the podinfo-values.yaml contains:
  co.elastic.logs/enabled: "true"
  co.elastic.logs/json.keys_under_root: "false"
  co.elastic.logs/json.message_key: message
  co.elastic.logs/json.overwrite_keys: "true"
  1. Install logstash via helm with the below config helm upgrade --install logstash elastic/logstash -f logstash-values.yaml, the content of logstash-values.yaml below.
fullnameOverride: logstash
  logstash.yml: |
    pipeline.ecs_compatibility: "v1"
    config.reload.automatic: true
    config.reload.interval: 900s

  logstash.conf: |
    input {
      beats {
        port => 5044
    filter {
      # don't need these
      if [kubernetes][namespace] == "kube-system" {
        drop { }
      # only care about podinfo
      if [kubernetes][container][name] in ["filebeat", "logstash"] {
        drop { }
    output {
      stdout { }
  1. Install filebeat via helm using helm upgrade --install filebeat ./helm-charts/filebeat -f filebeat-values.yaml --set imageTag=7.15.0 where filebeat-values.yaml contains:
fullnameOverride: filebeat
    co.elastic.logs/enabled: "false"
  enabled: true
    filebeat.yml: |
      # hints based autodiscover from the kubernetes API
          - type: kubernetes
            node: ${NODE_NAME}
            hints.enabled: true
              enabled: true
              type: container
                - /var/log/containers/*${}.log

      # output to logstash
        enabled: true
        hosts: ["logstash:5044"]
        loadbalance: true

    co.elastic.logs/enabled: "false"
  enabled: false

Logs for podinfo don't come through to logstash, just sits there idle. Filebeat generates a metric "bucketload" of logs since it is parsing its own multiline logs and feeding back on itself (meaning the hint to disable it doesn't seem to work).

Anyone got any suggestions? We've gone through many different permutations of configurations and sofar haven't found a single one that works.

BTW: podinfo generated logs look like the below as per: kubectl logs

{"level":"info","ts":"2021-10-19T02:02:02.563Z","caller":"podinfo/main.go:151","msg":"Starting podinfo","version":"6.0.0","revision":"","port":"9898"}
{"level":"info","ts":"2021-10-19T02:02:02.566Z","caller":"api/server.go:252","msg":"Starting HTTP Server.","addr":":9898"}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.