# Can't retrieve Nginx logs with Elastic Stask on ECK

**URL:** <https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 28, 2020, 2:42pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921 "2020-07-28T14:42:19Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [July 28, 2020, 2:42pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921/1 "2020-07-28T14:42:19Z")

</div>

Hello,

As a test, I want to retrieve **access.log** and **error.log** from a **Nginx** server with :

Filebeat ==\> Logstash ==\> Elasticsearch ==\> Kibana

Here's the situation:

- Windows 10 PRO with WSL1/Ubuntu18.04/Terminator and Docker Desktop

- Everything works under **ECK**. With the following files, all pods are **running 1/1**.

- To generate logs with Nginx, I just do **F5 or Ctrl+F5** on the **Welcome to nginx** page.

- The **autodiscover configuration** I propose in the **filebeat.yaml file** is not reliable, but I don't see where the problem comes from. Depending on the changes in the filebeat.yaml file, I retrieve, **at best** , data from the namespace beats (from Elasticsearch, Filebeat itself...) **but never access.log or error.log data from Nginx**. With the following files, here is what I get if I check my Elasticsearch indices:

- Sometimes, data with the **nginx\_test** tag is found in Kibana but never the **error** or **access** tags.

- If it helps, here's what I get when I check the state of the Kubernetes objects after starting the stack:

...

---

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [July 28, 2020, 2:43pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921/2 "2020-07-28T14:43:36Z")

</div>

_The rest of my message_

Here are the files I use:

### nginx.yaml:

```
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  namespace: beats
  labels:
    app: my-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: my-nginx

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  namespace: beats
spec:
  selector:
    matchLabels:
      app: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: my-nginx
    spec:
      containers:
        - name: my-nginx
          image: nginx
          ports:
          - containerPort: 80
          volumeMounts:
            - mountPath: "/var/log/nginx"
              name: nginx-data
      volumes:
        - name: nginx-data
          persistentVolumeClaim:
            claimName: nginx-data-pvc

```

### filebeat.yaml:

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: beats
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-

    tags: ["nginx_test"]

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true

          templates:
            - conditions.and:
                - equals.kubernetes.pod.name: nginx
                - contains.kubernetes.namespace: beats
              config:
                - module: nginx
                  access:
                    enabled: true
                    var.paths: ["/c/PATH/TO/PERSISTENT/VOLUME/nginx-data/access.log"]
                    subPath: access.log
                    tags: ["access"]

                  error:
                    enabled: true
                    var.paths: ["/c/PATH/TO/PERSISTENT/VOLUME/nginx-data/error.log"]
                    subPath: error.log
                    tags: ["error"]

    processors:

      - add_cloud_metadata:
      - add_kubernetes_metadata:
      - add_host_metadata:
      - add_docker_metadata:

    output.logstash:
      hosts: ["logstash:5044"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: beats
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.8.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: elasticsearch-es-http
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: ELASTICSEARCH_USERNAME
              value: elastic
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: elastic
                  name: elasticsearch-es-elastic-user
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true

      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: beats
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: beats
  labels:
    k8s-app: filebeat
---

```

### logstash.yaml:

```
---
apiVersion: v1
kind: Service
metadata:
  namespace: beats
  labels:
    app: logstash
  name: logstash
spec:
  ports:
    - name: "25826"
      port: 25826
      targetPort: 25826
    - name: "5044"
      port: 5044
      targetPort: 5044
  selector:
    app: logstash
status:
  loadBalancer: {}

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: beats
  name: logstash-configmap
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }

    filter {
    }

    output {
      if "nginx_test" in [tags] {
        elasticsearch {
          index => "nginx_test-%{[@metadata][beat]}-%{+YYYY.MM.dd-H.m}"
          hosts => ["${ES_HOSTS}"]
          user => "${ES_USER}"
          password => "${ES_PASSWORD}"
          cacert => '/etc/logstash/certificates/ca.crt'
        }
      }
    }

---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: logstash
  name: logstash
  namespace: beats
spec:
  containers:
    - image: docker.elastic.co/logstash/logstash:7.8.0
      name: logstash
      ports:
        - containerPort: 25826
        - containerPort: 5044
      env:
        - name: ES_HOSTS
          value: "https://elasticsearch-es-http:9200"
        - name: ES_USER
          value: "elastic"
        - name: ES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: elasticsearch-es-elastic-user
              key: elastic
      resources: {}
      volumeMounts:
        - name: config-volume
          mountPath: /usr/share/logstash/config
        - name: logstash-pipeline-volume
          mountPath: /usr/share/logstash/pipeline
        - name: cert-ca
          mountPath: "/etc/logstash/certificates"
          readOnly: true
  restartPolicy: OnFailure
  volumes:
    - name: config-volume
      configMap:
        name: logstash-configmap
        items:
          - key: logstash.yml
            path: logstash.yml
    - name: logstash-pipeline-volume
      configMap:
        name: logstash-configmap
        items:
          - key: logstash.conf
            path: logstash.conf
    - name: cert-ca
      secret:
        secretName: elasticsearch-es-http-certs-public
status: {}

```

### elasticsearch.yaml:

```
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: beats
spec:
  version: 7.8.0

  nodeSets:
    - name: elasticsearch
      count: 1
      config:
        node.store.allow_mmap: false
        node.master: true
        node.data: true
        node.ingest: true
        xpack.security.authc:
          anonymous:
            username: anonymous
            roles: superuser
            authz_exception: false
      podTemplate:
        metadata:
          labels:
            app: elasticsearch
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
              command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 4Gi
                  cpu: 0.5
                limits:
                  memory: 4Gi
                  cpu: 1
              env:
                - name: ES_JAVA_OPTS
                  value: "-Xms2g -Xmx2g"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            storageClassName: es-data
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 5Gi

```

### kibana.yaml:

```
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: beats
spec:
  version: 7.8.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  http:
    service:
      spec:
        type: LoadBalancer

```

### volume.yaml:

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: es-data
  namespace: beats
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nginx-data
  namespace: beats
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-pv
  namespace: beats
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: es-data
  hostPath:
    path: /c/PATH/TO/es-data

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-data-pv
  namespace: beats
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  #storageClassName: nginx-data
  storageClassName: ""
  hostPath:
    path: /c/PATH/TO/nginx-data

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-data-pvc
  namespace: beats
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: nginx-data-pv

```

Don't hesitate to ask me for more information and if you have any ideas to unblock me, thank you in advance!

Guillaume.

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [July 28, 2020, 3:24pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921/3 "2020-07-28T15:24:44Z")

</div>

First of all you should make sure that filebeat picks up these logs and passes them correctly further. Did you check this stage?

---

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [July 28, 2020, 3:43pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921/4 "2020-07-28T15:43:45Z")

</div>

Hello Marcin and thank you for your answer,

What is certain is that the Nginx logs do not pass (I can't find the error and access tags in Kibana). For the rest, it's variable... In fact, yesterday I was retrieving data from Elasticsearch, Filebeat, Kibana... from namespace beats but **nothing from Nginx**. This morning, after updating Docker Desktop + rebooting the PC, I have nothing left with the same code snippets (this is why I sais my code is not reliable).

As a result, I tried with something like this instead of the autodiscover feature:

```
filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

```

In this case, I get a very large amount of data (800.000+ entries), but nothing from Nginx. The PC also starts ventilating a lot and for a long time.

I'm a bit lost so I try a lot of things but I don't remember everything I tested.

---

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [July 28, 2020, 5:25pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921/5 "2020-07-28T17:25:35Z")

</div>

I just made a last test for tonight and without having modified anything I recovered data again (like yesterday) but nothing about Nginx.  
I don't know if this answers Marcin's question, but here is a series of screenshots made from what I've just recovered in Kibana:

### kubernetes.container.image:

![Capture_container_image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af19447b626ae9faefe52872d874f6a884142b3f.jpeg)

### kubernetes.container.name:

![Capture_container_name](https://us1.discourse-cdn.com/elastic/original/3X/0/9/091245e1a4a968cfe83da85c17ca2306ceaf55bc.jpeg)

### kubernetes.namespace:

![Capture_namespace](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6fe03c31bbee85d1ed9a18ce11935cff9434d0a7.jpeg)

### kubernetes.pod.name:

![Capture_pod_name](https://us1.discourse-cdn.com/elastic/original/3X/4/1/418a77cacaefcc23dc96d45591eaef3b7df7fe19.jpeg)

### log.file.path:

![Capture_path_file](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d4702de811d73003301e94ae7f0e061ccdac807.jpeg)

### message:

![Capture_message](https://us1.discourse-cdn.com/elastic/original/3X/a/f/afc521cdd3f4fab6e81bbfb98b039ba10a880eb8.jpeg)

### service.type:

![Capture_service](https://us1.discourse-cdn.com/elastic/original/3X/9/4/94d5ffac6be853c7fd1cf326e7fea98516233a8a.jpeg)

### tags:

![Capture_tag](https://us1.discourse-cdn.com/elastic/original/3X/e/a/ea6427dcea8eed9bfc70cd233ac9b96e3d9477aa.jpeg)

Anyway, there's no trace of Nginx... 🤔

---

<div class="post-metadata">

**Author:** ![LomigFR](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@LomigFR](https://discuss.elastic.co/u/LomigFR)\
**Post date:** [August 2, 2020, 4:08pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921/6 "2020-08-02T16:08:53Z")

</div>

Hi,

I'm coming back here to post a link where you can find the filebeat.yaml and volume.yaml files with which I solved my problem with Nginx's access.log and error.log data... if that helps anyone.

> [@Finaliser la récupération de logs Nginx via la stack Elastic/ECK](https://discuss.elastic.co/t/finaliser-la-recuperation-de-logs-nginx-via-la-stack-elastic-eck/242034/11):
>
> Bonjour, Je reviens vers vous avec un peu de nouveau. Après modification du fichier filebeat.yaml, je parviens à récupérer les fichiers access.log et error.log dans le container filebeat (j'ai ajouté un volume et un mountVolume dans le fichier de configuration, voir plus bas). Le souci est que ces fichiers restent désespérément vides malgré des F5/Ctrl+F5 sur la page d'accueil Welcome to nginx. Par ailleurs, j'ai modifié volontairement à la main, le fichier access.log dans le conteneur fi…

Happy reading to you.

Guillaume.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2020, 6:08pm UTC](https://discuss.elastic.co/t/cant-retrieve-nginx-logs-with-elastic-stask-on-eck/242921/7 "2020-08-30T18:08:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.