Packetbeat deployed using ECK crashes

I have started elastic 7.13 cluster using ECK 1.6.0, packetbeat pods keep crashing with the following error:

ERROR instance/beat.go:989 Exiting: sniffer loop failed: Sniffing error: Read Error
Exiting: sniffer loop failed: Sniffing error: Read Error

This is hosted using Amazon EKS 1.20.4 and the OS is bottlerocket 1.1.1

    apiVersion: beat.k8s.elastic.co/v1beta1
    kind: Beat
    metadata:
      name: elastic-pb
      namespace: elasticsearch
    spec:
      type: packetbeat
      version: 7.13.0
      elasticsearchRef:
        name: elastic
      kibanaRef:
        name: elastic
      config:
        logging.level: warning
        monitoring.enabled: true
        packetbeat:
          flows.enabled: false
          interfaces.device: any
          protocols:
          - type: dns
            ports: [ 53 ]
            include_authorities: true
            include_additionals: true
          - type: http
            ports: [ 80, 8080 ]
          - type: tls
            ports: [ 443, 8443 ]
          - type: mysql
            ports: [ 3306 ]
          - type: pgsql
            ports: [ 5432 ]
          - type: redis
            ports: [ 6379 ]
        processors:
        - add_cloud_metadata: {}
        - add_host_metadata: {}
        - add_kubernetes_metadata:
            host: ${HOSTNAME}
            indexers:
            - ip_port:
            matchers:
            - field_format:
                format: '%{[ip]}:%{[port]}'
      daemonSet:
        podTemplate:
          spec:
            serviceAccountName: packetbeat
            automountServiceAccountToken: true
            terminationGracePeriodSeconds: 30
            dnsPolicy: ClusterFirstWithHostNet
            hostNetwork: true # Allows to provide richer host metadata
            containers:
            - name: packetbeat
              resources:
                limits:
                  memory: 300Mi
                requests:
                  cpu: 100m
                  memory: 200Mi
              securityContext:
                runAsUser: 0
                capabilities:
                  add:
                    - NET_ADMIN
              volumeMounts:
              - name: data
                mountPath: /usr/share/packetbeat/data
            volumes:
            - name: data
              hostPath:
                path: /var/lib/packetbeat-data
                type: DirectoryOrCreate
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: packetbeat
    rules:
    - apiGroups: [""] # "" indicates the core API group
      resources:
      - namespaces
      - services
      - pods
      - nodes
      verbs:
      - get
      - watch
      - list
    - apiGroups: ["apps"]
      resources:
        - replicasets
      verbs: ["get", "list", "watch"]
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: packetbeat
      namespace: elasticsearch
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: packetbeat
    subjects:
    - kind: ServiceAccount
      name: packetbeat
      namespace: elasticsearch
    roleRef:
      kind: ClusterRole
      name: packetbeat
      apiGroup: rbac.authorization.k8s.io

Can I get help to know why this is happening and how to solve that?
Thank you very much

Any help would be appreciated :slight_smile:

Sorry, I'm not able to reproduce. I used the exact same resources on EKS 1.20.4 running on Bottlerocket and it seems to be working as expected:

  • Bottlerocket OS 1.1.1
  • containerd://1.4.4+bottlerocket
  • Kubelet v1.20.6
% k get es,kb,beats,pods  
NAME                                                 HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch.elasticsearch.k8s.elastic.co/elastic   green    3       7.13.0    Ready   13m

NAME                                   HEALTH   NODES   VERSION   AGE
kibana.kibana.k8s.elastic.co/elastic   green    1       7.13.0    13m

NAME                                  HEALTH   AVAILABLE   EXPECTED   TYPE         VERSION   AGE
beat.beat.k8s.elastic.co/elastic-pb   green    3           3          packetbeat   7.13.0    13m

NAME                                   READY   STATUS    RESTARTS   AGE
pod/elastic-es-default-0               1/1     Running   0          13m
pod/elastic-es-default-1               1/1     Running   0          13m
pod/elastic-es-default-2               1/1     Running   0          13m
pod/elastic-kb-5c86cf69b9-lxc6h        1/1     Running   0          9m55s
pod/elastic-pb-beat-packetbeat-66lt9   1/1     Running   1          9m53s
pod/elastic-pb-beat-packetbeat-9q555   1/1     Running   1          9m42s
pod/elastic-pb-beat-packetbeat-f4999   1/1     Running   2          9m35s
> GET _cat/indices/packetbeat-7.13.0
green open packetbeat-7.13.0-2021.06.14-000001 VNbpgNP_QM6UTvNzNpkanQ 1 1 5568 0 7.8mb 3.8mb

Anything you forgot to mention so I can try to reproduce again ?

Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.