# Competitive Filebeat access on local disk

**URL:** <https://discuss.elastic.co/t/competitive-filebeat-access-on-local-disk/243708>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [August 4, 2020, 12:58pm UTC](https://discuss.elastic.co/t/competitive-filebeat-access-on-local-disk/243708 "2020-08-04T12:58:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [August 4, 2020, 12:58pm UTC](https://discuss.elastic.co/t/competitive-filebeat-access-on-local-disk/243708/1 "2020-08-04T12:58:03Z")

</div>

Hello,

Following my previous posts, I'm coming to the end of my project. To finalize all this, I would have liked this time to pass the logs from 2 Nginx (located in different namespaces) to ES then Kibana (both located in a 3rd namespace).

With only one Nginx in the namespace\_a and ES/Kibana in the namespace\_esk : no problem... it works.  
On the other hand, if I launch the last namespace (with its own Nginx, Filebeat, Logstash), this new Filebeat crashes with this message :

Following my previous posts, I'm coming to the end of my project. To finalize all this, I would have liked this time to pass the logs from 2 Nginx (located in different namespaces) to ES then Kibana (both located in a 3rd namespace).

With only one Nginx in the namespace\_a and ES/Kibana in the namespace\_esk : no problem... it works.  
On the other hand, if I launch the last namespace (with its own Nginx, FIlebeat, Logstash), this new Filebeat crashes with this message :

```auto
2020-08-04T10:38:39.183Z	INFO	instance/beat.go:404	filebeat stopped.
2020-08-04T10:38:39.184Z	ERROR	instance/beat.go:958	Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

```

The namespace\_b also works very well if it is started alone with the namespace\_esk... so the problem appears well when both namespaces with Filebeat are started together.  
The problem apparently comes from a competitive access on the Nginx PV path but I don't know if it's possible to handle this.

Here are my files:

### filebeat.yaml (namespace\_a):

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: nsa
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    tags: ["nsa"]

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false

    filebeat.autodiscover:

      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true

          templates:
            - conditions.and:
                - contains.kubernetes.container.image: nginx
                - equals.kubernetes.namespace: nsa
              config:
                - module: nginx
                  access:
                    enabled: true
                    var.paths: ["/usr/share/filebeat/nginxlogs/access.log"]
                  error:
                    enabled: true
                    var.paths: ["/usr/share/filebeat/nginxlogs/error.log"]

    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - add_docker_metadata:

    output.logstash:
      hosts: ["logstash-nsa:5044"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: nsa
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.8.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: elasticsearch-es-http.esk
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: ELASTICSEARCH_USERNAME
              value: elastic
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: elastic
                  name: elasticsearch-es-elastic-user
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: es-certs
              mountPath: /mnt/elastic/tls.crt
              readOnly: true
              subPath: tls.crt
            - name: nginxlogs
              mountPath: /usr/share/filebeat/nginxlogs

      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate
        - name: es-certs
          secret:
            secretName: elasticsearch-es-http-certs-public
        - name: nginxlogs
          hostPath:
            path: /c/nsa/nginx-data

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: nsa
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: nsa
  labels:
    k8s-app: filebeat

```

### volume.yaml (namespace\_a):

```
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-data-pv-nsa
  namespace: nsa
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  hostPath:
    path: /c/nsa/nginx-data

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-data-pvc-nsa
  namespace: nsa
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: nginx-data-pv-nsa

```

### filebeat.yaml (namespace\_b):

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: nsb
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    tags: ["nsb"]

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false

    filebeat.autodiscover:

      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true

          templates:
            - conditions.and:
                - contains.kubernetes.container.image: nginx
                - equals.kubernetes.namespace: nsb
              config:
                - module: nginx
                  access:
                    enabled: true
                    var.paths: ["/usr/share/filebeat/nginxlogs/access.log"]
                  error:
                    enabled: true
                    var.paths: ["/usr/share/filebeat/nginxlogs/error.log"]

    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - add_docker_metadata:

    output.logstash:
      hosts: ["logstash-nsb:5044"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: nsb
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.8.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: elasticsearch-es-http.esk
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: ELASTICSEARCH_USERNAME
              value: elastic
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: elastic
                  name: elasticsearch-es-elastic-user
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: es-certs
              mountPath: /mnt/elastic/tls.crt
              readOnly: true
              subPath: tls.crt
            - name: nginxlogs
              mountPath: /usr/share/filebeat/nginxlogs

      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate
        - name: es-certs
          secret:
            secretName: elasticsearch-es-http-certs-public
        - name: nginxlogs
          hostPath:
            path: /c/nsb/nginx-data

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: nsb
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: nsb
  labels:
    k8s-app: filebeat

```

### volume.yaml (namespace\_b):

```
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-data-pv-nsb
  namespace: nsb
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  hostPath:
    path: /c/nsb/nginx-data

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-data-pvc-nsb
  namespace: nsb
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: nginx-data-pv-nsb

```

...

---

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [August 4, 2020, 12:58pm UTC](https://discuss.elastic.co/t/competitive-filebeat-access-on-local-disk/243708/2 "2020-08-04T12:58:14Z")

</div>

...

I changed the accessModes fields from ReadWriteOnce to **ReadWriteMany** but it didn't change anything.

For your information, I **work locally on a laptop**. I don't have access to a real infrastructure.

I tried my luck by adding a simple USB key to my laptop and then allowing it to be shared in Docker Desktop. The directory corresponding to the desired PV is well created there, but Filebeat crashes always occur if another instance of this Beats is already running (as explained above).

Is there a way to make both namespaces (with 1 filebeat each) work without this competitive access problem on the "/"? There must be some kind of rights issues: is there a solution with ECK to handle this kind of situation, please?  
Could the problem be related to Docker Desktop itself?

Thank you in advance for your help,

Guillaume.

---

<div class="post-metadata">

**Author:** ![michael.morello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael.morello/32/47448_2.png) [@michael.morello](https://discuss.elastic.co/u/michael.morello)\
**Post date:** [August 5, 2020, 9:08am UTC](https://discuss.elastic.co/t/competitive-filebeat-access-on-local-disk/243708/3 "2020-08-05T09:08:49Z")

</div>

The 2 Filebeat instances are sharing the same data directory, which I guess is `/var/lib/filebeat-data` . I think this is the root cause of the error.

I would first try to deploy only one filebeat instance for all your namespaces. This is the approach and the architecture we promote through [the Beats support in ECK](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-beat-quickstart.html#k8s-beat-quickstart).

---

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [August 5, 2020, 9:43am UTC](https://discuss.elastic.co/t/competitive-filebeat-access-on-local-disk/243708/4 "2020-08-05T09:43:03Z")

</div>

Hi,

In fact, after some research of my own, I have resolved the concern in question. As you specify in your answer, the original problem is related to the (non-permitted) sharing of the **/var/lib/filebeat-data** directory.

So here are the changes made in the two filebeat files. With this, I have two Filebeat components that work in separate namespaces (one harvests Nginx logs and the other harvests Apache logs). See the added and modified lines below.

### filebeat.yaml (namespace\_a):

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: nsa
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    tags: ["nsa"]

    path.data: ${path.home}/data-a ### <<=== Added line

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false

    filebeat.autodiscover:

      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true

          templates:
            - conditions.and:
                - contains.kubernetes.container.image: nginx
                - equals.kubernetes.namespace: nsa
              config:
                - module: nginx
                  access:
                    enabled: true
                    var.paths: ["/usr/share/filebeat/nginxlogs/access.log"]
                  error:
                    enabled: true
                    var.paths: ["/usr/share/filebeat/nginxlogs/error.log"]

    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - add_docker_metadata:

    output.logstash:
      hosts: ["logstash-nsa:5044"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: nsa
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.8.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: elasticsearch-es-http.esk
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: ELASTICSEARCH_USERNAME
              value: elastic
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: elastic
                  name: elasticsearch-es-elastic-user
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data-a ### <<=== Modified line
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: es-certs
              mountPath: /mnt/elastic/tls.crt
              readOnly: true
              subPath: tls.crt
            - name: nginxlogs
              mountPath: /usr/share/filebeat/nginxlogs

      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data-a ### <<=== Modified line
            type: DirectoryOrCreate
        - name: es-certs
          secret:
            secretName: elasticsearch-es-http-certs-public
        - name: nginxlogs
          hostPath:
            path: /c/nsa/nginx-data

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: nsa
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: nsa
  labels:
    k8s-app: filebeat

```

### filebeat.yaml (namespace\_b):

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: nsb
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    tags: ["nsb"]

    path.data: ${path.home}/data-b ### <<=== Added line

    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false

    filebeat.autodiscover:

      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true

          templates:
            - conditions.and:
                - contains.kubernetes.container.image: httpd
                - equals.kubernetes.namespace: nsb
              config:
                - module: apache2
                  access:
                    input:
                      type: docker
                      containers.ids:
                        - ${data.kubernetes.container.id}
                  error:
                    input:
                      type: docker
                      containers.ids:
                        - ${data.kubernetes.container.id}

    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - add_docker_metadata:

    output.logstash:
      hosts: ["logstash-nsb:5044"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: nsb
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.8.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: elasticsearch-es-http.esk
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: ELASTICSEARCH_USERNAME
              value: elastic
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: elastic
                  name: elasticsearch-es-elastic-user
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data-b ### <<=== Modified line
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: es-certs
              mountPath: /mnt/elastic/tls.crt
              readOnly: true
              subPath: tls.crt

      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data-b ### <<=== Modified line
            type: DirectoryOrCreate
        - name: es-certs
          secret:
            secretName: elasticsearch-es-http-certs-public

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: nsb
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: nsb
  labels:
    k8s-app: filebeat

---

```

Thank you again for your response!

Guillaume.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 8:08am UTC](https://discuss.elastic.co/t/competitive-filebeat-access-on-local-disk/243708/5 "2022-11-04T08:08:48Z")

</div>