How to add podip on add_kubernetes_metadata processor?

Elastic Stack Beats
1

Hi I'm running filebeat daemonset on my k8s cluster
have try different config scenario but do not get the pod ip data
here is my config

  processors:
    - add_kubernetes_metadata:
        default_indexers.enabled: true
        default_matchers.enabled: true
        indexers:
          - pod_uid:

am i miss something?

2

Hi @dadayoo, welcome to discuss :slightly_smiling_face:

What filebeat configuration are you using? Take a look to the reference manifests provided in beats: beats/filebeat-kubernetes.yaml at v7.12.0 · elastic/beats · GitHub

Specifically to the add_kubernetes_metadata example. Something like this should work (if your logs are under /var/log/containers:

        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Or alternativelly you can use autodiscover as commented in the same reference config. Autodiscover already fills in the kubernetes metadata and then add_kubernetes_metadata is not needed.

3

hi @jsoriano thanks for your replay
here is my filebeat config

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"


output.console:
  pretty: true

here is console output

{
  "@timestamp": "2021-04-01T11:38:37.351Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.0"
  },
  "ecs": {
    "version": "1.8.0"
  },
  "host": {
    "name": "filebeat-new-6d87884ccf-797vm"
  },
  "agent": {
    "ephemeral_id": "519086ef-c4d0-4b12-84a7-5feea55af52c",
    "id": "8fca3548-29b3-4510-883b-9fba7cd82cce",
    "name": "filebeat-new-6d87884ccf-797vm",
    "type": "filebeat",
    "version": "7.12.0",
    "hostname": "filebeat-new-6d87884ccf-797vm"
  },
  "log": {
    "offset": 27567571,
    "file": {
      "path": "/var/lib/kubelet/pods/e5e497e1-8bdb-45b9-9141-c3ca2dce8eb8/volumes/kubernetes.io~empty-dir/logs/applogs/resource-manage/tomcat_stdout.log"
    }
  },
  "message": "application stdout message",
  "input": {
    "type": "log"
  }
}

still didn't get podip here :sleepy:

4

Ok, I think the problem is with the path, I guess that these are not logs from stdout/stdin, but from a mounted volume, and is under the pods path, not the containers path.

Try something like the example in these docs to match with the pod logs path: Add Kubernetes metadata | Filebeat Reference [7.12] | Elastic

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - pod_uid:
      matchers:
        - logs_path:
            logs_path: '/var/lib/kubelet/pods'
            resource_type: 'pod'

Or you can try using autodiscover instead of filebeat.inputs, that wouldn't need add_kubernetes_metadata. It would be something like this (not tested, review the condition and the path):

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.pod.name: tomcat
          config:
            - type: log
              paths:
                - /var/lib/kubelet/pods/${kubernetes.pod.uid}*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log
5

yes, file beat collect logs from Hostpath which is kubernetes emptyDir mount point
inner pod path:

/home/sclogs/logs/applogs/myapp/tomcat_stdout.log

Host path:

/var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

I have try

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log

  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - pod_uid:
      matchers:
        - logs_path:
            logs_path: '/var/lib/kubelet/pods'
            resource_type: 'pod'
     
output.console:
  pretty: true

console output:

{
  "@timestamp": "2021-04-01T12:21:03.795Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.0"
  },
  "message": "here is application stdout message",
  "input": {
    "type": "log"
  },
  "kubernetes": {
    "labels": {
      "app": "workapp",
      "pod-template-hash": "645bc8fb7b",
      "szone": "sc"
    },
    "node": {
      "name": "node.172.28.131.146",
      "uid": "5f187da8-4650-4fcc-b4ea-61dccfc1410a",
      "labels": {
        "kubernetes_io/os": "linux",
        "beta_kubernetes_io/arch": "amd64",
        "beta_kubernetes_io/os": "linux"
      },
      "hostname": "node.172.28.131.146"
    },
    "namespace_uid": "da7ac4c7-f733-4728-988e-7a5ba6e2dedc",
    "pod": {
      "name": "workapp-645bc8fb7b-qqf5p",
      "uid": "961e07e0-11b3-40c8-92ea-0be804d5d949"
    },
    "namespace": "sc",
    "replicaset": {
      "name": "workapp-645bc8fb7b"
    }
  },
  "ecs": {
    "version": "1.8.0"
  },
  "host": {
    "name": "filebeat-new-6d87884ccf-whszl"
  },
  "agent": {
    "hostname": "filebeat-new-6d87884ccf-whszl",
    "ephemeral_id": "f9bf1229-039b-4921-b4ea-933133286903",
    "id": "d8fcd0ac-5522-4319-9425-289bb5ddf13d",
    "name": "filebeat-new-6d87884ccf-whszl",
    "type": "filebeat",
    "version": "7.12.0"
  },
  "log": {
    "offset": 1944254,
    "file": {
      "path": "/var/lib/kubelet/pods/961e07e0-11b3-40c8-92ea-0be804d5d949/volumes/kubernetes.io~empty-dir/logs/applogs/worknum/tomcat_stdout.log"
    }
  }
}

more filed get ,but still no podip
I have try autodicover, but it get "Non-zero metric"

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.pod.name: tomcat
          config:
            - type: log
              paths:
                - /var/lib/kubelet/pods/${kubernetes.pod.uid}*/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log


output.console:
  pretty: true
6

Umm, I thought that pod IP was included in metadata, but not, it seems it is only included in the data (as data.host) that can be used in autodiscover templates. When collecting metrics, the endpoint uses to be also available in the event, but I see now that it is not included in log events.

I have created an issue to follow on this: Add pod IP to kubernetes metadata · Issue #24902 · elastic/beats · GitHub

As a possible workaround, as this IP is available in autodiscover templates, if you get a working configuration maybe you can fill the field with a processor, something like this:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.pod.name: workapp
          config:
            - type: log
              paths:
                - /var/lib/kubelet/pods/${data.kubernetes.pod.uid}/volumes/kubernetes.io~empty-dir/logs/applogs/*/tomcat_stdout.log
              processors:
              - add_fields:
                  target: kubernetes.pod
                  fields:
                    ip: "${data.host}"

output.console:
  pretty: true

Notice that I have changed some things:

  • Adapted the condition to match your pod, review it if you want to match a different set of pods.
  • Fixed variable used in path in config, they have to be prefixed by data..
  • Added add_fields processor to add kubernetes.pod.ip with the value of data.host.
7

@jsoriano
Thanks for help!
I turn to autodiscover and it works! finally got podip :yum: :+1: :+1:
hope I can get podip from events metadata on next version of filebeat :stuck_out_tongue_winking_eye:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.