Packetbeat Flows and Kubernetes not working fine

[rikatz (Ricardo Katz)]

I've been trying to use Packetbeat to map my Kubernetes Cluster traffic without success.

The idea here is to use Packetbeat to map all the flows from the Cluster and store them into Elasticsearch, enriching those events with Kubernetes PODs Metadata (namespace and name, mainly).

But when trying to do this, no metadata is inserted. The following is my packetbeat.yml config:

packetbeat.interfaces.device: any
output.elasticsearch:
  hosts: ["REDACTED:9200"]
setup.kibana:
  host: "REDACTED:5601"
packetbeat.interfaces.device: any
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 100
packetbeat.flows:
  timeout: 30s
  period: 10s
  enabled: true
  processors:
  - add_kubernetes_metadata:
      host: REDACTED
      in_cluster: false
      kube_config: /etc/kubernetes/node-kubeconfig.yaml

When trying to start packetbeat, I get the following messages:

|2018-09-24T16:53:35.937-0300|INFO|kubernetes/watcher.go:180|kubernetes: Performing a resource sync for *v1.PodList|
|---|---|---|---|
|2018-09-24T16:53:35.955-0300|INFO|kubernetes/watcher.go:194|kubernetes: Resource sync done|
|2018-09-24T16:53:35.955-0300|INFO|kubernetes/watcher.go:238|kubernetes: Watching API for resource events|

But the event only stand as the following JSON (no Metadata):

{
  "_index": "packetbeat-6.4.1-2018.09.24",
  "_type": "doc",
  "_id": "UiAoDWYB5pf0lEE4guaU",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-09-24T19:57:30.000Z",
    "transport": "tcp",
    "source": {
      "ip": "10.79.REDACTED",
      "port": 56386,
      "stats": {
        "net_bytes_total": 296,
        "net_packets_total": 4
      },
      "mac": "72:dd:61:ad:eb:67"
    },
    "flow_id": "EQQA////DP//////FP8BAAFy3WGt62fu7u7u7u4KTxiqrBDYlkLc0gQ",
    "beat": {
      "version": "6.4.1",
      "name": "REDACTED",
      "hostname": "REDACTED"
    },
    "final": false,
    "dest": {
      "mac": "ee:ee:ee:ee:ee:ee",
      "ip": "172.16.216.150",
      "port": 1234
    },
    "start_time": "2018-09-24T19:57:18.090Z",
    "last_time": "2018-09-24T19:57:25.218Z",
    "type": "flow",
    "host": {
      "name": "REDACTED"
    }
  },
  "fields": {
    "start_time": [
      "2018-09-24T19:57:18.090Z"
    ],
    "@timestamp": [
      "2018-09-24T19:57:30.000Z"
    ],
    "last_time": [
      "2018-09-24T19:57:25.218Z"
    ]
  },
  "highlight": {
    "dest.port": [
      "@kibana-highlighted-field@1234@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1537819050000
  ]
}

Any idea how to improve this?

EDIT: Got the following message when enabling Debug "kubernetes":

2018-09-24T17:13:10.007-0300	DEBUG	[kubernetes]	add_kubernetes_metadata/matchers.go:167	Unable to apply field format pattern on event

Thanks!

[exekias (Carlos Pérez Aradros)]

Hi @rikatz,

Packetbeat is configured to enrich events based on ip & port fields. I see in your event that you would be interested on using source.ip and source.port this time. Probably something like this could work:

  - add_kubernetes_metadata:
      host: REDACTED
      in_cluster: false
      kube_config: /etc/kubernetes/node-kubeconfig.yaml
      matchers:
        - field_format:
            format: '%{[source.ip]}:%{[source.port]}'

(Take into account that ports need to be declared in the Pods so Packetbeat can enrich based on that info)

Let us know how it goes! we may consider adding these as defaults