Add_kubernetes_metadata processor

Elastic Stack Beats
1

Hi,

We are setting up packetbeat in kubernetes cluster. I'm following https://github.com/elastic/examples/blob/master/MonitoringKubernetes/packetbeat-kubernetes.yaml

Packetbeat kubernetes processor is not adding kubernetes metadata. We want logs to have kubernetes pod-name and namespace.

How can I add pod-name and namespace to the logs?

Below is the packetbeat.yml

    setup.dashboards.enabled: true
    setup.template.enabled: true
    setup.template.settings:
      index.number_of_shards: 2
    packetbeat.interfaces.device: any
    packetbeat.protocols.http:
      ports: [80, 443, 8080, 8000, 5000, 8002, 4010, 4030, 8983]
      hide_keywords: ['pass', 'password', 'passwd']
      send_headers: ["host", "User-Agent", "X-App-Build-Version", "X-Default-City", "X-Os-Version", "X-Phone-Manufacturer", "X-Phone-Platform", "X-App-Version", "X-Access-Token"]
      include_body_for: ["application/json", "application/x-www-form-urlencoded"]
      real_ip_header: "X-Forwarded-For"
      send_request: true
      send_response: true
    processors:
      - add_kubernetes_metadata:
    cloud.auth: 'elastic:pwd'
    cloud.id: 'cloudiddd'
    output.elasticsearch:
      hosts: ['host:9243']
      username: "elastic"
      password: "pwdd"

following is the packetbeat document:

{
  "_index": "packetbeat-6.5.2-2018.12.10",
  "_type": "doc",
  "_id": "rUTdl2cBCHzkHuUeELXw",
  "_score": 1,
  "_source": {
    "@timestamp": "2018-12-10T11:25:05.536Z",
    "client_ip": "100.108.111.133",
    "ip": "100.108.111.176",
    "path": "/ping",
    "status": "OK",
    "beat": {
      "name": "ip-172-20-72-199",
      "hostname": "ip-172-20-72-199",
      "version": "6.5.2"
    },
    "client_port": 38682,
    "type": "http",
    "client_proc": "",
    "bytes_in": 1528,
    "bytes_out": 548,
    "proc": "",
    "request": "GET /ping HTTP/1.1\r\nhost: pranay321-inventory-web.dev.abc.in\r\naccept: text/plain\r\naccept-encoding: gzip, deflate, br\r\naccept-language: en-US,en;q=0.9\r\ncookie: _ga=GA1.2.1450467738.1513414415; WZRK_G=ea3ad5b9a21f4d8cba8381d78391d881; mp_2fc3d1bb8a38efd412d0f763790d9998_mixpanel=%7B%22distinct_id%22%3A%20%22pranay.sankpal%40abc.in%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; intercom-lou-zgbmsle7=1; _hp2_id.4023131627=%7B%22userId%22%3A%225557349102867226%22%2C%22pageviewId%22%3A%220394380536054322%22%2C%22sessionId%22%3A%224552244222923878%22%2C%22identity%22%3A%22pranay.sankpal%abc.in%22%2C%22trackerVersion%22%3A%224.0%22%2C%22identityField%22%3A%22email%22%2C%22isIdentified%22%3A1%2C%22oldIdentity%22%3Anull%7D; __insp_wid=1458088866; __insp_slim=1538820427833; __insp_nv=true; __insp_targlpu=aHR0cHM6Ly9zdGFnaW5nLW9vYy5waGFybWVhc3kuaW4vbG9naW4%2FbmV4dD0vcngtYXZhaWxhYmlsaXR5; __insp_targlpt=UGhhcm1FYXN5IC0gT3JkZXIgb24gQ2FsbA%3D%3D; __insp_norec_sess=true; cto_lwid=6f7a5c75-1ae4-4c20-b61b-f5a9a312becf\r\nif-none-match: W/\"95784973cc639977bff9361970016850\"\r\nsave-data: on\r\nuser-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36\r\nx-forwarded-for: 52.76.109.129\r\nx-forwarded-port: 443\r\nx-forwarded-proto: https\r\nx-request-id: 674312fd-1d38-45f4-a8bd-9611cf2d65c2\r\nx-envoy-expected-rq-timeout-ms: 3000\r\nx-envoy-original-path: /ping\r\ncontent-length: 0\r\n\r\n",
    "client_server": "",
    "port": 80,
    "http": {
      "request": {
        "headers": {
          "content-length": 0,
          "host": "pranay321-inventory-web.dev.abc.in",
          "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
        },
        "params": ""
      },
      "response": {
        "phrase": "Not Modified",
        "headers": {
          "content-length": 0
        },
        "code": 304
      }
    },
    "host": {
      "name": "ip-172-20-72-199"
    },
    "query": "GET /ping",
    "real_ip": "110.16.109.129",
    "server": "",
    "response": "HTTP/1.1 304 Not Modified\r\nServer: nginx/1.15.7\r\nDate: Mon, 10 Dec 2018 11:25:05 GMT\r\nConnection: keep-alive\r\nAccess-Control-Allow-Origin: \r\nAccess-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS\r\nAccess-Control-Allow-Headers: Origin, Content-Type, Authorization, Accept, X-Requested-With\r\nAccess-Control-Max-Age: 1728000\r\nAccess-Control-Allow-Credentials: true\r\nETag: W/\"95784973cc639977bff9361970016850\"\r\nCache-Control: max-age=0, private, must-revalidate\r\nX-Request-Id: 674312fd-1d38-45f4-a8bd-9611cf2d65c2\r\nX-Runtime: 0.002927\r\n\r\n",
    "responsetime": 4,
    "method": "GET"
  },
  "fields": {
    "@timestamp": [
      "2018-12-10T11:25:05.536Z"
    ]
  }
}
2

You need to set the host from ${HOSTNAME}:

processors:
 - add_kubernetes_metadata:
    host: ${HOSTNAME}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.