Filebeat:Not able to scrap logs of Spark on Kubernetes

[cpawali (Chandrashekhar C Pawali)]

I am running spark on Kubernetes . Need to send spark driver logs to elastic through filebeat.
And I need Driver pod details also in my logs.Below are my configuration

filebeat_configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      paths:
        - /var/log/containers/spark-pi-v1-driver*.log
      symlinks: true
      fields:
        component: "spark_driver"
        clusterName: "odp_luffy"
      close_renamed: true
      close_removed: true
      close_timeout: "30m"
      tail_files: true
      
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      processors:
      - add_kubernetes_metadata:
          labels.dedot: true
          annotations.dedot: true


    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch.logging}:${ELASTICSEARCH_PORT:9200}']
      index: "cpawali-1"
      username: 'elastic'
      password: 'changeme'
  
    setup.template.name: "spark-pi-%{[agent.version]}"
    setup.template.pattern: "spark-pi-%{[agent.version]}"

    logging.level: debug

filebeat.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: elastic/filebeat:8.11.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch.logging
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

Its not sending anyof the kubernetes metadata, i tried with many other add_kubernetes_metadata config.
Here is my pod details which i need filebeat to send along with logs