Kubernetes hints auto-discover: specify parser by stream (filestream input)

Elastic Stack Beats
1

Hi. I'm trying to have filebeat use a different parser depending on stream of the event (stdout/stderr).

This is for ingress-nginx, which has the ability to customize the access log into json, but not error logs, for some reasons. So I want to parse the access_logs (on stdout) with ndjson, and the error logs with container.

My filebeat config is this way:

 filebeat.autodiscover:
  providers:
  - type: kubernetes
    hints:
      enabled: true
      default_config:
        type: filestream
        id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
        take_over: true
        enabled: false
        paths:
        - /var/log/containers/*-${data.kubernetes.container.id}.log  # CRI path
        parsers:
        - container: 
            stream: all
            format: auto
        prospector:
          scanner:
            symlinks: true
    add_ressource_metadata: # Considers namespace annotations for hints
      deployment: false
      cronjob: false
      namespace:
        include_annotations:
          - "nsannotations1"

# processors and outputs, not relevant

And I use the following annotations on the pods:

metadata:
  annotations:
    co.elastic.logs/enabled: "true"
    co.elastic.logs/json.add_error_key: "true"
    co.elastic.logs/json.target: ingress

What I would like to obtain is a dynamic configuration looking something like that, I guess:

        type: filestream
        id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
        paths:
        - /var/log/containers/*-${data.kubernetes.container.id}.log  # CRI path
        parsers:
        - ndjson:
             stream: stdout
             target: ingress
             add_error_key: true
        - container: 
            stream: stderr
            format: auto

However it's not clear how I should format my hints to achieve this, or if it's possible at all in fact :thinking:

I know I could use processors for that, or include/exclude_lines, but using the stream as differentiator seems both more elegant and likely to perform better.
Does anyone had achieved something like this ?

2

I couldn't find any info on the performance difference between using the ndjson parser or a decode_json_fields processor, by the way.