How to add podip on add_kubernetes_metadata processor?

dadayoo · April 1, 2021, 9:25am

Hi I'm running filebeat daemonset on my k8s cluster
have try different config scenario but do not get the pod ip data
here is my config

  processors:
    - add_kubernetes_metadata:
        default_indexers.enabled: true
        default_matchers.enabled: true
        indexers:
          - pod_uid:

am i miss something?

jsoriano · April 1, 2021, 9:51am

Hi @dadayoo, welcome to discuss

What filebeat configuration are you using? Take a look to the reference manifests provided in beats: beats/filebeat-kubernetes.yaml at v7.12.0 · elastic/beats · GitHub

Specifically to the add_kubernetes_metadata example. Something like this should work (if your logs are under /var/log/containers:

        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Or alternativelly you can use autodiscover as commented in the same reference config. Autodiscover already fills in the kubernetes metadata and then add_kubernetes_metadata is not needed.

dadayoo · April 1, 2021, 11:45am

hi @jsoriano thanks for your replay
here is my filebeat config

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"


output.console:
  pretty: true

here is console output

{
  "@timestamp": "2021-04-01T11:38:37.351Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.0"
  },
  "ecs": {
    "version": "1.8.0"
  },
  "host": {
    "name": "filebeat-new-6d87884ccf-797vm"
  },
  "agent": {
    "ephemeral_id": "519086ef-c4d0-4b12-84a7-5feea55af52c",
    "id": "8fca3548-29b3-4510-883b-9fba7cd82cce",
    "name": "filebeat-new-6d87884ccf-797vm",
    "type": "filebeat",
    "version": "7.12.0",
    "hostname": "filebeat-new-6d87884ccf-797vm"
  },
  "log": {
    "offset": 27567571,
    "file": {
      "path": "/var/lib/kubelet/pods/e5e497e1-8bdb-45b9-9141-c3ca2dce8eb8/volumes/kubernetes.io~empty-dir/logs/applogs/resource-manage/tomcat_stdout.log"
    }
  },
  "message": "application stdout message",
  "input": {
    "type": "log"
  }
}

still didn't get podip here

jsoriano · April 1, 2021, 12:03pm

Ok, I think the problem is with the path, I guess that these are not logs from stdout/stdin, but from a mounted volume, and is under the pods path, not the containers path.

Try something like the example in these docs to match with the pod logs path: Add Kubernetes metadata | Filebeat Reference [7.12] | Elastic

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - pod_uid:
      matchers:
        - logs_path:
            logs_path: '/var/lib/kubelet/pods'
            resource_type: 'pod'

Or you can try using autodiscover instead of filebeat.inputs, that wouldn't need add_kubernetes_metadata. It would be something like this (not tested, review the condition and the path):

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.pod.name: tomcat
          config:
            - type: log
              paths:
                - /var/lib/kubelet/pods/${kubernetes.pod.uid}*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

dadayoo · April 1, 2021, 12:41pm

yes, file beat collect logs from Hostpath which is kubernetes emptyDir mount point
inner pod path:

/home/sclogs/logs/applogs/myapp/tomcat_stdout.log

Host path:

/var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

I have try

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - pod_uid:
      matchers:
        - logs_path:
            logs_path: '/var/lib/kubelet/pods'
            resource_type: 'pod'
     
output.console:
  pretty: true

console output:

{
  "@timestamp": "2021-04-01T12:21:03.795Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.0"
  },
  "message": "here is application stdout message",
  "input": {
    "type": "log"
  },
  "kubernetes": {
    "labels": {
      "app": "workapp",
      "pod-template-hash": "645bc8fb7b",
      "szone": "sc"
    },
    "node": {
      "name": "node.172.28.131.146",
      "uid": "5f187da8-4650-4fcc-b4ea-61dccfc1410a",
      "labels": {
        "kubernetes_io/os": "linux",
        "beta_kubernetes_io/arch": "amd64",
        "beta_kubernetes_io/os": "linux"
      },
      "hostname": "node.172.28.131.146"
    },
    "namespace_uid": "da7ac4c7-f733-4728-988e-7a5ba6e2dedc",
    "pod": {
      "name": "workapp-645bc8fb7b-qqf5p",
      "uid": "961e07e0-11b3-40c8-92ea-0be804d5d949"
    },
    "namespace": "sc",
    "replicaset": {
      "name": "workapp-645bc8fb7b"
    }
  },
  "ecs": {
    "version": "1.8.0"
  },
  "host": {
    "name": "filebeat-new-6d87884ccf-whszl"
  },
  "agent": {
    "hostname": "filebeat-new-6d87884ccf-whszl",
    "ephemeral_id": "f9bf1229-039b-4921-b4ea-933133286903",
    "id": "d8fcd0ac-5522-4319-9425-289bb5ddf13d",
    "name": "filebeat-new-6d87884ccf-whszl",
    "type": "filebeat",
    "version": "7.12.0"
  },
  "log": {
    "offset": 1944254,
    "file": {
      "path": "/var/lib/kubelet/pods/961e07e0-11b3-40c8-92ea-0be804d5d949/volumes/kubernetes.io~empty-dir/logs/applogs/worknum/tomcat_stdout.log"
    }
  }
}

more filed get ,but still no podip
I have try autodicover, but it get "Non-zero metric"

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.pod.name: tomcat
          config:
            - type: log
              paths:
                - /var/lib/kubelet/pods/${kubernetes.pod.uid}*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log


output.console:
  pretty: true

jsoriano · April 1, 2021, 2:07pm

Umm, I thought that pod IP was included in metadata, but not, it seems it is only included in the data (as data.host) that can be used in autodiscover templates. When collecting metrics, the endpoint uses to be also available in the event, but I see now that it is not included in log events.

I have created an issue to follow on this: Add pod IP to kubernetes metadata · Issue #24902 · elastic/beats · GitHub

As a possible workaround, as this IP is available in autodiscover templates, if you get a working configuration maybe you can fill the field with a processor, something like this:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.pod.name: workapp
          config:
            - type: log
              paths:
                - /var/lib/kubelet/pods/${data.kubernetes.pod.uid}/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log
              processors:
              - add_fields:
                  target: kubernetes.pod
                  fields:
                    ip: "${data.host}"

output.console:
  pretty: true

Notice that I have changed some things:

Adapted the condition to match your pod, review it if you want to match a different set of pods.
Fixed variable used in path in config, they have to be prefixed by data..
Added add_fields processor to add kubernetes.pod.ip with the value of data.host.

dadayoo · April 2, 2021, 7:45am

@jsoriano
Thanks for help!
I turn to autodiscover and it works! finally got podip
hope I can get podip from events metadata on next version of filebeat

system · April 30, 2021, 9:45am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Send kubernetes pod IP address data from filebeat Beats filebeat	2	575	October 23, 2019
Kubernetes metadata missing with add_kubernetes_metadata enabled Beats filebeat	7	2771	December 4, 2020
Kubernetes_metadata does not work on Kubernetes Windows Beats filebeat	3	646	September 23, 2020
Kubernetes metadata not showing (add_kubernetes_metadata) Beats filebeat	6	624	January 11, 2021
Add_kubernetes_metadata processor does not seem to work. Filebeat Data indexed to Elastic does not have any fields relevant to kubernetes Elasticsearch docker	3	695	February 18, 2023

How to add podip on add_kubernetes_metadata processor?

Related Topics