# Filebeat:Not able to scrap logs of Spark on Kubernetes

**URL:** <https://discuss.elastic.co/t/filebeat-not-able-to-scrap-logs-of-spark-on-kubernetes/365851>\
**Category:** Beats\
**Tags:** elastic-stack-monitoring, docker, filebeat\
**Created:** [August 31, 2024, 10:46am UTC](https://discuss.elastic.co/t/filebeat-not-able-to-scrap-logs-of-spark-on-kubernetes/365851 "2024-08-31T10:46:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cpawali](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cpawali/32/126350_2.png) [@cpawali](https://discuss.elastic.co/u/cpawali)\
**Post date:** [August 31, 2024, 10:46am UTC](https://discuss.elastic.co/t/filebeat-not-able-to-scrap-logs-of-spark-on-kubernetes/365851/1 "2024-08-31T10:46:37Z")

</div>

I am running spark on Kubernetes . Need to send spark driver logs to elastic through filebeat.  
And I need Driver pod details also in my logs.Below are my configuration

**filebeat\_configmap.yaml**

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      paths:
        - /var/log/containers/spark-pi-v1-driver*.log
      symlinks: true
      fields:
        component: "spark_driver"
        clusterName: "odp_luffy"
      close_renamed: true
      close_removed: true
      close_timeout: "30m"
      tail_files: true
      
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      processors:
      - add_kubernetes_metadata:
          labels.dedot: true
          annotations.dedot: true

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch.logging}:${ELASTICSEARCH_PORT:9200}']
      index: "cpawali-1"
      username: 'elastic'
      password: 'changeme'
  
    setup.template.name: "spark-pi-%{[agent.version]}"
    setup.template.pattern: "spark-pi-%{[agent.version]}"

    logging.level: debug

```

**filebeat.yaml**

```auto
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: elastic/filebeat:8.11.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch.logging
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

```

Its not sending anyof the kubernetes metadata, i tried with many other **add\_kubernetes\_metadata** config.  
Here is my pod details which i need filebeat to send along with logs

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/0/90220d802ac7ce86178a6daef6f614974d4db580.png)

---

<div class="post-metadata">

**Author:** ![cpawali](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cpawali/32/126350_2.png) [@cpawali](https://discuss.elastic.co/u/cpawali)\
**Post date:** [August 31, 2024, 10:50am UTC](https://discuss.elastic.co/t/filebeat-not-able-to-scrap-logs-of-spark-on-kubernetes/365851/2 "2024-08-31T10:50:44Z")

</div>

From #Elasticsearch to #Beats

---

<div class="post-metadata">

**Author:** ![cpawali](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cpawali/32/126350_2.png) [@cpawali](https://discuss.elastic.co/u/cpawali)\
**Post date:** [August 31, 2024, 10:50am UTC](https://discuss.elastic.co/t/filebeat-not-able-to-scrap-logs-of-spark-on-kubernetes/365851/3 "2024-08-31T10:50:44Z")

</div>

Added #filebeat