# Kubernetes hints auto-discover: specify parser by stream (filestream input)

**URL:** <https://discuss.elastic.co/t/kubernetes-hints-auto-discover-specify-parser-by-stream-filestream-input/360080>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 23, 2024, 1:39pm UTC](https://discuss.elastic.co/t/kubernetes-hints-auto-discover-specify-parser-by-stream-filestream-input/360080 "2024-05-23T13:39:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![VannTen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vannten/32/132551_2.png) [@VannTen](https://discuss.elastic.co/u/VannTen)\
**Post date:** [May 23, 2024, 1:39pm UTC](https://discuss.elastic.co/t/kubernetes-hints-auto-discover-specify-parser-by-stream-filestream-input/360080/1 "2024-05-23T13:39:31Z")

</div>

Hi. I'm trying to have filebeat use a different parser depending on stream of the event (stdout/stderr).

This is for ingress-nginx, which has the ability to customize the access log into json, but not error logs, for some reasons. So I want to parse the access\_logs (on stdout) with ndjson, and the error logs with container.

My filebeat config is this way:

```yml
 filebeat.autodiscover:
  providers:
  - type: kubernetes
    hints:
      enabled: true
      default_config:
        type: filestream
        id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
        take_over: true
        enabled: false
        paths:
        - /var/log/containers/*-${data.kubernetes.container.id}.log # CRI path
        parsers:
        - container: 
            stream: all
            format: auto
        prospector:
          scanner:
            symlinks: true
    add_ressource_metadata: # Considers namespace annotations for hints
      deployment: false
      cronjob: false
      namespace:
        include_annotations:
          - "nsannotations1"

# processors and outputs, not relevant

```

And I use the following annotations on the pods:

```yml
metadata:
  annotations:
    co.elastic.logs/enabled: "true"
    co.elastic.logs/json.add_error_key: "true"
    co.elastic.logs/json.target: ingress

```

What I would like to obtain is a dynamic configuration looking something like that, I guess:

```yml
        type: filestream
        id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
        paths:
        - /var/log/containers/*-${data.kubernetes.container.id}.log # CRI path
        parsers:
        - ndjson:
             stream: stdout
             target: ingress
             add_error_key: true
        - container: 
            stream: stderr
            format: auto

```

However it's not clear how I should format my hints to achieve this, or if it's possible at all in fact 🤔

I know I could use processors for that, or include/exclude\_lines, but using the stream as differentiator seems both more elegant and likely to perform better.  
Does anyone had achieved something like this ?

---

<div class="post-metadata">

**Author:** ![VannTen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vannten/32/132551_2.png) [@VannTen](https://discuss.elastic.co/u/VannTen)\
**Post date:** [May 24, 2024, 12:54pm UTC](https://discuss.elastic.co/t/kubernetes-hints-auto-discover-specify-parser-by-stream-filestream-input/360080/2 "2024-05-24T12:54:39Z")

</div>

I couldn't find any info on the performance difference between using the ndjson parser or a `decode_json_fields` processor, by the way.