Problem to update to filebeat 7.7.0 and parser nginx-ingress-controller on Kubernetes

[David_Oceans]

Hi!

I was using Kubernetes with this version of Filebeat: docker.elastic.co/beats/filebeat:7.3.2

In order to use the new functionalities to parser nginx-ingress-controller I've update to docker.elastic.co/beats/filebeat:7.7.0

But with my configuration, only changing the image I have some errors.

2020-05-13T14:26:25.084Z	ERROR	[kubernetes]	add_kubernetes_metadata/matchers.go:91	Error extracting container id - source value does not contain matcher's logs_path '/var/lib/docker/containers/'.

I read that was removing the support for add_kubernetes_metadata
https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-7.7.0.html

So... I don't know how I should configure my filebeat. I'm using this configuration, I should replace by another thing?

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
      multiline.negate: false
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Thank you very much

[David_Oceans]

Hi,

I tried updating 7.7.0 and removing this part an the errors disappear but I'm not sure if I have to change it instead of removing.

By the other hand, to parser nginx-ingress-controller I don't have the logs in files

kubectl exec -ti nginx-ingress-controller-5c64888b48-h5mxp -n ingress-controller bash
bash-5.0$ ls -l /var/log/nginx/
total 0
lrwxrwxrwx    1 www-data www-data        11 Feb 24 12:47 access.log -> /dev/stdout
lrwxrwxrwx    1 www-data www-data        11 Feb 24 12:47 error.log -> /dev/stderr

I've tried with and without path, but doesn't log nothing in Kibana

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
          templates:
            - condition:
                equals:
                  kubernetes.labels.app: nginx-ingress
              config:
                - module: nginx
                  ingress_controller:
                    enabled: true
                    var.paths: ["/var/log//nginx/*.log"]

The condition I think is correct, right?

Name:           nginx-ingress-controller-5c64888b48-h5mxp
Namespace:      ingress-controller
Priority:       0
Node:           gke-apps-stage-default-node-pool-d001bfd8-tfpq/10.31.0.21
Start Time:     Mon, 23 Mar 2020 12:38:08 +0100
Labels:         app=nginx-ingress
                app.kubernetes.io/component=controller
                pod-template-hash=5c64888b48
                release=nginx-ingress

I don't know where is the error, I think is about the path, but what do you think?

Thank you very much

[jsoriano (Jaime Soriano)]

Hey @David_Oceans,

Take into account that filebeat runs in the host, so it may not have access to the files in the container (like these /var/log/nginx/*.log). What one usually does is to use a container input, that collects logs from stdout/stderr.
You have some examples in the documentation: https://www.elastic.co/guide/en/beats/filebeat/7.7/configuration-autodiscover.html

In your case it would be something like this:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
          templates:
            - condition:
                equals:
                  kubernetes.labels.app: nginx-ingress
              config:
                - module: nginx
                  ingress_controller:
                    enabled: true
                    input:
                      type: container
                      paths:
                        - /var/log/containers/*-${data.kubernetes.container.id}.log

Alternativelly you can use hints-based autodiscover, without templates:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true

And add annotations like these ones to your ingress controller pods:

  co.elastic.logs/module: nginx
  co.elastic.logs/fileset.stdout: ingress_controller
  co.elastic.logs/fileset.stderr: error

[tolausson (Thomas Olausson)]

I'm having a similar issue. It seems that the error is coming from that the default logs_path is picked up (rather than the configured one).

Notice how the error mentions /var/lib/docker/containers,
but the matcher's logs_path is /var/log/containers
This default can be found in the code here

I have not been able to solve it myself.

[rhallier]

Well, I'm running into the same problem. It seems that the matcher logs_path is not picked up, only the default value (/var/lib/docker/containers)
My version of Filebeat is 7.7.1
Regards

[ChrsMark (Chris Mark)]

Hi everyone!

I would suggest upgrading to 7.8 since this seems to be a regression until 7.7.1.

C.