Filebeat autodiscover doesn't pick-up inputs

Elastic Stack Beats
1

Hi there!

I've deployed Filebeat in the "observability" namespace (with all roles, bindings being set to observability - not sure if this is ok). It starts without issues, but it doesn't discover any inputs.

This is my configuration. I'm new to this so I suspect there's something wrong with it, but I don't know what (documentation seems all over the place)

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: filebeat-config
      namespace: observability
      labels:
        k8s-app: filebeat
    data:
      filebeat.yml: |-
        setup.ilm.enabled: false
    
        filebeat.autodiscover:
            providers:
            - type: kubernetes
              node: ${NODE_NAME}
              templates:
                - config:
                    input:
                      - type: container
                        tail_files: true
                        paths:
                          - /var/log/containers/${kubernetes.container.id}/*.log
                        fields:
                          appname: ${kubernetes.service.name}
                          environment: ${kubernetes.namespace}

        processors:
          - add_cloud_metadata:
          - add_host_metadata:
          - drop_fields:
              fields: ["beat.name", "beat.hostname", "beat.version", "beat.os"]

          - decode_json_fields:
              fields: ["message"]
              max_depth: 8
              target: ""
              overwrite_keys: true

        output.elasticsearch:  
          hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
          indices:
            - index: "%{[fields.appname]}_%{[fields.environment]}_logs-%{+yyyy.MM.dd}" 
---

Any support is much appreciated!

2

Hi!

Here https://github.com/elastic/beats/blob/5501ce848afca10590696ba1f4bb7426660ebec8/deploy/kubernetes/filebeat-kubernetes.yaml#L23 you can find an out-of-the box manifest that will work in most cases.

C.

3

Thanks Chris. That's where I got my start, but I wanted to control how indices were created and I couldn't figure out how to do that with the out of the box version - I couldn't find if and how I can extract more data from matching, like I could from autodiscover (and such I could add extra fields that were used in the output section).

4

I see an extra input key in your config that I don't think should be there could you try to remove it?

Also in order to go from the provided example to your specific case please try step by step and verify each step/addition/modification works before moving the final big configuration which is hard to troubleshoot.

5

Thanks, that was a good spot. I've re-read the docs and I've made some progress:

    filebeat.autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - config:
                - type: container
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}/*.log
                  fields:
                    appname: ${data.kubernetes.service.name}
                    environment: ${data.kubernetes.namespace}

Now I'm getting something like:

2020-10-27T15:02:43.523Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*-10c3cfbf1dd9be7845c604466ae6ff626d257294908aca42c377f201c8f1e98a/*.log]
2020-10-27T15:02:43.523Z	INFO	input/input.go:114	Starting input of type: container; ID: 9603579383151076722
2020-10-27T15:02:43.523Z	ERROR	[autodiscover]	cfgfile/list.go:96	Error creating runner from config: No paths were defined for input accessing config

Will keep digging.

6

Well, I figured out there was a small typo with the path expression (after manually logging into the node and looking at the logs). I could never get fields to work, which was messing with the index creation, but luckily I had all the info in the kubernetes object, so I used that. The docs should state that you kinda need to enable symlinks in order for the whole thing to work, otherwise files will be skipped (could not understand initially why some worked and others didn't, until I started Filebeat with debug logs)

Anyway, this is the config that works for me:

   setup:
      ilm.enabled: false
      template:
        name: "kytelogs"
        pattern: "kytelogs-*"

    filebeat.autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals: 
                  kubernetes.namespace: sandbox
              config:
                - type: container
                  tail_files: true
                  symlinks: true
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

      - decode_json_fields:
          fields: ["message"]
          max_depth: 8
          target: ""
          overwrite_keys: true

    output.elasticsearch:  
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      index: "kytelogs-%{[kubernetes.labels.app]}-%{[kubernetes.labels.domain]}-%{[agent.version]}-%{+yyyy.MM.dd}"
7

Hi @CatalinM!

Thanks for the input here! I'm afraid enabling symlinks is not the common case for all environments and hence it wouldn't help to have it mentioned in the docs since it could be misleading. Checking the logs should be of help in such cases as happened with your case.

Thanks!

8

Hi @ChrsMark. I won't contradict you on how common this scenarios is - it's not really my specialty. Thanks for your help though, it really got me going.

For anyone having issues with this, it's key to start the Filebeat container with debug logs. That will show you a lot of things going wrong (like the symlinks, like how it couldn't push events - which I then traced to a failing substitution in the index name).

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.