Kubernetes - ConfigMap

[bitva77 (Matt Hemingway)]

Hi! I'm having an issue with Filebeat running in Kubernetes.

I'm using the filebeat-kubernetes.yml file from the documentation and have made following change to the filebeat-prospectors ConfigMap:

> data:
>   kubernetes.yml: |-
>     - type: docker
>       containers.ids:
>         - "*"
>       processors:
>       - add_kubernetes_metadata:
>           in_cluster: true
>       filebeat.autodiscover:
>         providers:
>           - type: kubernetes
>             templates:
>               - condition:
>                   equals:
>                     kubernetes.namespace: development
>                 config:
>                   - type: docker
>                     containers.ids:
>                       - "${data.kubernetes.container.id}"

The pods are erroring with this message:

Unable to unpack config file due to error: missing field accessing '0.filebeat' (source:'/usr/share/filebeat/prospectors.d/kubernetes.yml')

Any help as to where I went wrong would be greatly appreciated

Thanks!

-Matt

[exekias (Carlos Pérez Aradros)]

Hi @bitva77,

You are mixing a prospector configuration with autodiscover (which is a top level thing). If you plan to use autodiscover only you can drop prospectors.yml and put autodiscover settings in the main filebeat.yml. I created a sample gist doing so (disclaimer: I didn't test it):

https://gist.github.com/exekias/f8b0b300966fa148ac239ffaa70b5721

Best regards

[bitva77 (Matt Hemingway)]

OH! Thanks! I apparently have a hard time understanding things lol

Took a bit of massaging but here's a working config if anyone else is googling like I did:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                equals:
                  kubernetes.namespace: development
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"

Let me ask you, when running in K8s, why wouldn't you use autodiscover?

Thanks again!

[exekias (Carlos Pérez Aradros)]

We released it pretty recently, at some point it should be the default in the manifests, once we consider it stable enough.

Your feedback is important here, too, we learn from what users report and request

[bitva77 (Matt Hemingway)]

cool beans. I'll just keep putting updates and stuff here then. I have a question below:

Our goal is to capture logs from our microservices and not all the crap that comes from K8s, so I've limited filebeats to the namespaces we care about and so far so good with that.

We also send everything to Logstash instead. Here's what I got so far for the filebeat-config ConfigMap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                equals:
                  kubernetes.namespace: development
            - condition:
                equals:
                  kubernetes.namespace: sqa
            - condition:
                equals:
                  kubernetes.namespace: stage
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"

    output.logstash:
      hosts: ['${LOGSTASH_HOST}:${LOGSTASH_PORT}']

I pass the actual LOGSTASH_HOST and _PORT later on in the DaemonSet configuration.

I commented out the rest of the prospector references in the original YAML as well since we aren't using any of that.

Question: Is there a need for me to specify filebeat.registry_file ? and have that be a physical location on the host (via hostPath)?

If something happens to the filebeat pod and it gets restarted, it's going to resend all the same logs from the namespaces I've identified. To avoid that, I need that registry file to persist, correct?

[exekias (Carlos Pérez Aradros)]

You don't need to update registry_file setting, as it defaults to the data volume. It's recommended to mount the data volume as a hostPath, so we persist the resgistry file across pod changes. That will avoid us to send logs again on restart.

I recently added a warning about that here: https://github.com/elastic/beats/pull/6526

[bitva77 (Matt Hemingway)]

Gotcha. FYI for anyone googlin, to persist the data volume, change the data volume definition from emptyDir to this:

  - name: data
    hostPath:
      path: /var/lib/filebeat

of course you need the directory /var/lib/filebeat to exist on every node in your cluster and then the registry file should persist no matter what happens to the pods.

[bitva77 (Matt Hemingway)]

A few things now:

1. It appears only the last condition statement works. Meaning in the above configuration, only the "stage" namespace is being looked at.

2. Our logs are sent by the apps are multi-line but are not being sent by Filebeat via multi-line no matter what I try to make the multiline.pattern. Is that not observed with autodiscover?

Thanks again!

[bitva77 (Matt Hemingway)]

Alrighty, I figured out the multi-conditional statements to only look at certain namespaces with this configuration:

data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          in_cluster: true
          templates:
            - condition:
                or:
                  - equals:
                      kubernetes.namespace: development
                  - equals:
                      kubernetes.namespace: sqa
                  - equals:
                      kubernetes.namespace: stage
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"

I'm still having issues with getting multiline.pattern to work though...will keep at it until I pass out here. Any advice you can give me on that (when I awake from my coma) would be appreciated!

Here's a sample of the logs that our application is sending:

{"log":"\u001b[40m\u001b[32minfo\u001b[39m\u001b[22m\u001b[49m: Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor[1]\n","stream":"stdout","time":"2018-03-15T17:58:43.269647888Z"}
{"log":" Executing ObjectResult, writing value Microsoft.AspNetCore.Mvc.ControllerContext.\n","stream":"stdout","time":"2018-03-15T17:58:43.269673063Z"}

I have this under providers:

      multiline.pattern: '^\\u001b'
      multiline.negate: true
      multiline.match: after

Thanks!

[bitva77 (Matt Hemingway)]

I think I got it. All in the indentation:

data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          in_cluster: true
          templates:
            - condition:
                or:
                  - equals:
                      kubernetes.namespace: development
                  - equals:
                      kubernetes.namespace: sqa
                  - equals:
                      kubernetes.namespace: stage
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  multiline.pattern: '^[[:space:]]'
                  multiline.negate: false
                  multiline.match: after

    output.logstash:
      hosts: ['${LOGSTASH_HOST}:${LOGSTASH_PORT}']

sorry for all the back and forth with myself here. Just trying to document the steps as this autodiscover stuff seems pretty new.

Thanks!

-Matt