Filebeat [7.15.0] hints based discovery does nothing

[gwvandesteeg (Gerwin)]

Continuing the discussion from Filebeat doesn't send logs to Elasticsearch:

We're having the exact same problem as the above which doesn't appear to have ever been resolved.
Using filebeat 7.15.0, exact same config as above, running on kind, we have a very simple test setup.

• filebeat configured as above to write to logstash
• logstash configured to write to stdout
• podinfo installed

To reproduce:

1. Start kind (or whatever k8s cluster you want to use)
2. Install podinfo using helm3 helm upgrade --install --wait backend -f podinfo-values.yaml --set redis.enabled=true podinfo/podinfo where the podinfo-values.yaml contains:

---
podAnnotations:
  co.elastic.logs/enabled: "true"
  co.elastic.logs/json.keys_under_root: "false"
  co.elastic.logs/json.message_key: message
  co.elastic.logs/json.overwrite_keys: "true"

3. Install logstash via helm with the below config helm upgrade --install logstash elastic/logstash -f logstash-values.yaml, the content of logstash-values.yaml below.

---
fullnameOverride: logstash
logstashConfig:
  logstash.yml: |
    http.host: 0.0.0.0
    pipeline.ecs_compatibility: "v1"
    config.reload.automatic: true
    config.reload.interval: 900s

logstashPipeline:
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }
    filter {
      # don't need these
      if [kubernetes][namespace] == "kube-system" {
        drop { }
      }
      # only care about podinfo
      if [kubernetes][container][name] in ["filebeat", "logstash"] {
        drop { }
      }
    }
    output {
      stdout { }
    }

4. Install filebeat via helm using helm upgrade --install filebeat ./helm-charts/filebeat -f filebeat-values.yaml --set imageTag=7.15.0 where filebeat-values.yaml contains:

---
fullnameOverride: filebeat
daemonset:
  annotations:
    co.elastic.logs/enabled: "false"
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      # hints based autodiscover from the kubernetes API
      filebeat.autodiscover:
        providers:
          - type: kubernetes
            node: ${NODE_NAME}
            hints.enabled: true
            hints.default_config:
              enabled: true
              type: container
              paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log

      # output to logstash
      output.logstash:
        enabled: true
        hosts: ["logstash:5044"]
        loadbalance: true

deployment:
  annotations:
    co.elastic.logs/enabled: "false"
  enabled: false

Logs for podinfo don't come through to logstash, just sits there idle. Filebeat generates a metric "bucketload" of logs since it is parsing its own multiline logs and feeding back on itself (meaning the hint to disable it doesn't seem to work).

Anyone got any suggestions? We've gone through many different permutations of configurations and sofar haven't found a single one that works.

BTW: podinfo generated logs look like the below as per: kubectl logs

{"level":"info","ts":"2021-10-19T02:02:02.563Z","caller":"podinfo/main.go:151","msg":"Starting podinfo","version":"6.0.0","revision":"","port":"9898"}
{"level":"info","ts":"2021-10-19T02:02:02.566Z","caller":"api/server.go:252","msg":"Starting HTTP Server.","addr":":9898"}