Kubernetes autodiscover with hints, default/fallback newline from annotations

[havlan (Håvard Langdal)]

I was wondering if setting annotations with a newline and other options will override default config for that specific event. If a pod does not have those annotations, will it get default options from filebeat config file? Like a fallback or default value.

If pod A runs with these annotations.

co.elastic.logs/multiline.pattern: '^\['
co.elastic.logs/multiline.negate: 'true'
co.elastic.logs/multiline.match: after

But this is in the filebeat.yml

multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after

How will multiline pattern be handled?

[jsoriano (Jaime Soriano)]

Hi @havlan,

Autodiscover providers create new inputs that are independent of other inputs that could have been configured in the filebeat.yml file or by any other means, so in principle no fallback is done.

In the case of kubernetes hints-based autodiscovery a docker input is instantiated for any container in a pod with annotations, and this input instance only contains the container id, and the configuration provided via annotations. If a module is used, then a module is instantiated instead of a docker input, but also containing only the configuration provided on the annotations.

[havlan (Håvard Langdal)]

Hi @jsoriano and thanks for answering my questions.
If I wasn't clear enough, I meant the autodiscover feature.
My config is like this.

filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints_enabled: true
          templates:
            - condition:
                or:
                  - equals:
                      kubernetes.namespace: kube-system
                  - equals:
                      kubernetes.namespace: default
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
                  multiline.pattern: '^[[:space:]]'
                  multiline.negate: false
                  multiline.match: after
    processors:
      - add_kubernetes_metadata:
          in_cluster: true

    output.logstash:
      hosts: ['${LOGSTASH_HOSTS}']

So if I start a pod A with annotations they get used as config for the events from that pod?

annotations:
    co.elastic.logs/multiline.pattern: '^\['
    co.elastic.logs/multiline.negate: 'true'
    co.elastic.logs/multiline.match: after

And if I start pod B without annotations that uses the default from autodiscover config?

Edit: So if I want to use hints, I would need to specify (in this case) exclude_lines, multiline.pattern, multiline.negate and multiline.match for every pod on that node (of course if they are needed)?

Thanks again for answering.

[jsoriano (Jaime Soriano)]

Oh, I understand now

With the current implementation, hints are only considered if no template matches, so in your case, in principle, for pods in the kube-system and default namespaces the config in the template will be used and hints will be ignored. For the rest of pods, hints will be used if they exist.

One thing you can try to achieve is to add a condition to your template that excludes pods that contain the multiline pattern annotation.

[havlan (Håvard Langdal)]

Awesome. Thanks again for the clarification

[havlan (Håvard Langdal)]

How would this interact with renaming or extracting fields? Do I have to use annotations for this too or is it an option to force every event through adding the required fields (_service, _token and type in this case). Perhaps rename processor would work, for hint events too, but that would move important information from the kubernetes labels entries.

Example config:

filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints_enabled: true
          templates:
            - condition: # is this a list(?) https://github.com/elastic/beats/issues/6084#issuecomment-411011649
                and:
                  - or: 
                    - equals:
                        kubernetes.namespace: kube-system
                    - equals:
                        kubernetes.namespace: default
                  - not:
                      has_fields: ['kubernetes.annotations.co.elastic.logs/multiline.pattern'] # use the configuration that comes with the deployment manifest
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  # First option to pass metadata
                  fields:
                    _service: "${data.kubernetes.labels.ut-service}"
                    _token: "${data.kubernetes.labels.ut-token}"
                    type: "${data.kubernetes.labels.ut-type}"
                  fields_under_root: true
                  exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
                  multiline.pattern: '^[[:space:]]'
                  multiline.negate: false
                  multiline.match: after
    processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_fields:
          fields: ["host"]

      # This ensures that every log that passes has required fields.
      - drop_event.when.not:
          has_fields: ['kubernetes.labels.ut-service', 'kubernetes.labels.ut-token', 'kubernetes.labels.ut-type']
      
    output.logstash:
      hosts: ['localhost:5044']
      

[jsoriano (Jaime Soriano)]

When using autodiscover templates you can use any setting available for inputs, this includes fields. Variables will be replaced.
fields is not available when using hints.