# Use k8s provider fields in filebeat config

**URL:** <https://discuss.elastic.co/t/use-k8s-provider-fields-in-filebeat-config/346864>\
**Category:** Elastic Agent\
**Tags:** filebeat\
**Created:** [November 10, 2023, 11:08am UTC](https://discuss.elastic.co/t/use-k8s-provider-fields-in-filebeat-config/346864 "2023-11-10T11:08:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![GeorgeGkinis](https://avatars.discourse-cdn.com/v4/letter/g/ecc23a/32.png) [@GeorgeGkinis](https://discuss.elastic.co/u/GeorgeGkinis)\
**Post date:** [November 10, 2023, 11:08am UTC](https://discuss.elastic.co/t/use-k8s-provider-fields-in-filebeat-config/346864/1 "2023-11-10T11:08:25Z")

</div>

Hello everyone!

We are deploying the Elastic Agent as a daemonset to slurp up our container logs using hints based autodiscovery.

This works and we can selectively parse pods based on the following hint:

```auto
    podTemplate:
      metadata:
        annotations:
          co.elastic.hints/package: "container_logs_ecs"

```

The standalone agent.yml provider section:

```auto
...
providers:
  kubernetes:
    node: ${NODE_NAME}
    scope: node
    include_annotations: true
    include_labels: true
    hints:
      default_container_logs: false
      enabled: true
...

```

We mount the following file on the Agent pod through its configmap and it works fine:

```auto
# container_logs_ecs.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: external-inputs
  namespace: kube-system
  labels:
    k8s-app: elastic-agent
data:
  container_logs_ecs.yml: |-
    inputs:
      - name: hints-filestream-container-logs
        id: hints-filestream-container-logs-${kubernetes.hints.container_id}
        type: filestream
        use_output: default
        streams:
          - condition: ${kubernetes.hints.container_logs_ecs.enabled} == true
            data_stream:
              dataset: cps-test
              type: logs
            parsers:
              - container:
                  format: auto
                  stream: ${kubernetes.hints.container_logs.stream|'all'}
              - ndjson:
                  target: ""
                  ignore_decoding_error: false
                  expand_keys: true
                  add_error_key: true
            paths:
              - /var/log/containers/*${kubernetes.hints.container_id}.log
            prospector:
              scanner:
                symlinks: true
        data_stream.namespace: default

```

What we want to achieve is to send the logs to a different datastream based on its kubernetes fields by changing the dataset field:

```auto
dataset: cps-${data.kubernetes.namespace}

```

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: external-inputs
  namespace: kube-system
  labels:
    k8s-app: elastic-agent
data:
  container_logs_ecs.yml: |-
    inputs:
      - name: hints-filestream-container-logs
        id: hints-filestream-container-logs-${kubernetes.hints.container_id}
        type: filestream
        use_output: default
        streams:
          - condition: ${kubernetes.hints.container_logs_ecs.enabled} == true
            data_stream:
              dataset: cps-${data.kubernetes.namespace}
              type: logs
            parsers:
              - container:
                  format: auto
                  stream: ${kubernetes.hints.container_logs.stream|'all'}
              - ndjson:
                  target: ""
                  ignore_decoding_error: false
                  expand_keys: true
                  add_error_key: true
            paths:
              - /var/log/containers/*${kubernetes.hints.container_id}.log
            prospector:
              scanner:
                symlinks: true
        data_stream.namespace: default

```

The dynamic config then does not include configuration for this pod.

How can I access fields returned by the kubernetes provider?  
Is there somewhere a way to see all available fields under **kubernetes.hints.** \* or **data.** \*?

Thanks!

---

<div class="post-metadata">

**Author:** ![GeorgeGkinis](https://avatars.discourse-cdn.com/v4/letter/g/ecc23a/32.png) [@GeorgeGkinis](https://discuss.elastic.co/u/GeorgeGkinis)\
**Post date:** [November 13, 2023, 9:27am UTC](https://discuss.elastic.co/t/use-k8s-provider-fields-in-filebeat-config/346864/2 "2023-11-13T09:27:07Z")

</div>

I eventually switched to conditions based autodiscovery.  
All the kubernetes.\* fields are then available.

Is this maybe a bug in hints based autodiscovery?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 11, 2023, 9:27am UTC](https://discuss.elastic.co/t/use-k8s-provider-fields-in-filebeat-config/346864/3 "2023-12-11T09:27:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.