Kubernetes - Packetbeat implitation

Elastic Stack Beats
1

I am trying to implement packetbeat for specific services in Kubernetes (EKS cluster) but I am unable to get proper metrics.

Issues:

  1. Unable to get desired index name. the default index is packetbeat-7.8.1-2020.07.30-000001

  2. Kubernetes's metadata is not coming.

Packet_demonset.yaml

    ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: packetbeat-input-config
  namespace: kube-system
  labels:
    k8s-app: packetbeat-input
    kubernetes.io/cluster-service: "true"
data:
  packetbeat.yml: |-
    setup.dashboards.enabled: false
    setup.template.enabled: true
    setup.template.name: "packet-k8s"
    setup.template.pattern: "packet-k8s"
    setup.template.settings:
      index.number_of_shards: 2

    packetbeat.interfaces.device: any

    packetbeat.protocols:
    - type: dns
      ports: [53]
      include_authorities: true
      include_additionals: true

    - type: http
      ports: [80, 8080, 9200]

    packetbeat.flows:
      timeout: 30s
      period: 10s

    processors:
      - include_fields:
          fields: ["fields","tags","agent.name","http","type","event","query","url","method","kubernetes.namespace","kubernetes.pod.name","kubernetes.deployment.name"]
      - add_kubernetes_metadata:
          #default_indexers.enabled: false
          #default_matchers.enabled: false
          namespace: default
          indexers:
            - pod_name:
          matchers:
            - fields:
                lookup_fields: ['kubernetes.pod.name', 'kubernetes.deployment.name']
      - if:
          equals:
           kubernetes.deployment.name: "test-app"
        then:
          - decode_json_fields:
              when:
               equals:
                url.path: "/api/test/txn/confirm"
              fields: ["http.response.body.content"]
              target: "confirmResponse"
          - decode_json_fields:
              when:
               equals:
                url.path: "/api/test/txn/confirm"
              fields: ["http.request.body.content"] 
              target: "confirmRequest"
          - drop_fields:
              when:
               equals:
                url.path: "/api/test/txn/confirm"
              fields: ["http.response.body.content", "http.request.body.content"]
              ignore_missing: true
      - if:
          equals:
           kubernetes.deployment.name: "webapp"
        then:
          - drop_event:
              when:
               not:
                regexp:
                  url.path: "^/api/pay/v[0-4]/payment/(initiate|eligibility|pay)$"
          - decode_json_fields:
              when:
               not:
                equals:
                  http.response.status_code: 200
              fields: ["http.response.body.content"]
              target: "http.response.body.error_content"

             
    output.elasticsearch:
      hosts: ["10.235.247.33:9200"]
      index: "packet-k8s-%{[kubernetes.deployment.name]}-%{+yyyy.MM.dd}"
      username: "elastic"
      password: "password"
      
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: packetbeat-input
  namespace: kube-system
  labels:
    k8s-app: packetbeat-input
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: packetbeat-input
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: packetbeat-input
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      containers:
      - name: packetbeat-input
        image: docker.elastic.co/beats/packetbeat:7.8.1
        imagePullPolicy: Always
        args: [
          "-c", "/etc/packetbeat.yml",
          "-e",
        ]
        securityContext:
          runAsUser: 0
          capabilities:
            add:
            - NET_ADMIN
        volumeMounts:
        - name: config
          mountPath: /etc/packetbeat.yml
          readOnly: true
          subPath: packetbeat.yml
        - name: data
          mountPath: /usr/share/packetbeat/data
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: packetbeat-input-config
      - name: data
        emptyDir: {}
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: packetbeat-input
subjects:
- kind: ServiceAccount
  name: packetbeat-input
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: packetbeat-input
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: packetbeat-input
  labels:
    k8s-app: packetbeat-input
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: packetbeat-input
  namespace: kube-system
  labels:
    k8s-app: packetbeat-input
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.