Can anyone guide me to correct autodiscover condition?

Elastic Stack Beats
1

FB: 7.16.3

I try "- and:" and "and:" both get "missing or invalid condition".

filebeat.autodiscover.providers:
  - type: kubernetes
    node: ${NODE_NAME}
    templates:
      - condition:
           and: 
              - not:
                  kubernetes.container.name: "fluentd"
              - equals:
                  kubernetes.namespace: "kube-system"
        config:
          - module: nginx
            access:
              input:
                type: container
                stream: stdout
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log
            error:
              input:
                type: container
                stream: stderr
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log

update:
I found my mistake, there is no comparison condition under "not".

filebeat.autodiscover.providers:
  - type: kubernetes
    node: ${NODE_NAME}
    templates:
    - condition:
        and:
        - equals.kubernetes.namespace: "kube-system"
        - not.equals.kubernetes.container.name: "fluentd"

update:
In addition, I also found that under a logical operation, there can only be one comparison condition, for example, the following "and" can only use one "equals".

I'm not sure if my understanding is correct, but I test the following statement cannot run filebeat.
Statements are just for testing, do not make any evaluation of significance.

filebeat.autodiscover.providers:
  - type: kubernetes
    node: ${NODE_NAME}
    templates:
      - condition:
          and:
          - equals:
              kubernetes.namespace: "kube-system"
          - equals:
              kubernetes.namespace: "elastic"
          - not:
              - equals:
                  kubernetes.container.name: "fluentd"

filebeat.autodiscover.providers:
  - type: kubernetes
    node: ${NODE_NAME}
    templates:
      - condition:
          and:
          - or:
              - equals:
                  kubernetes.namespace: "kube-system"
              - equals:
                  kubernetes.namespace: "elastic"
         - not:
              - equals:
                  kubernetes.container.name: "fluentd"
2

The same problem occurs on output.kafka, the log says the given type is incorrect.

WARN [conditions] conditions/equals.go:48 expected string but got type []string in equals condition.

output.kafka:
   enabled: true
   hosts: ["1.1.1.1:9092"]
   topics:
   - topic: "logs_backend_dev"
       when:
         and:
         - equals:
             type: "backend"
         - equals:
             environment: "dev"
   parttion.round_robin:
      reacheble_only: false
   required_acks: 1
   commpression: gzip
   max_message_bytes: 1000000

@jsoriano Hello, I found that you have answered similar questions on github, can you help me answer this question again?

3

@wajika I see you made several updates of the original topic. Did you find a working configuration? what is the whole configuration you have now?

4

I marked the update: section in the first post, everything after that is new questions, and my problem has been unsolved because I couldn't find any correct multi-conditional syntax to test successfully.

I only use one condition now and it is running successfully, but I want to know how to write the correct multi-condition syntax. I have sorted out some syntax formats according to the results of the forum search, but there will be corresponding problems when using it on filebeat 7.16.3.

For example this prompt expected string but got type []string in equals condition. I also made a lot of changes but I did not keep records.

5

The not condition doesn't expect a list, but a single element Define processors | Filebeat Reference [8.3] | Elastic

Try this:

      - condition:
          and:
            - or:
              - equals:
                  kubernetes.namespace: "kube-system"
              - equals:
                  kubernetes.namespace: "elastic"
            - not:
                equals:
                  kubernetes.container.name: "fluentd"
6

Thanks, then there's nothing wrong with the not condition.
Another question, why does the following when.or not take effect? I didn't find the topic name "logs-hz" in kafka.

name: xxxx
tags: ["pre","dev","test"]
fields:
  env: development

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config.enabled: false

processors:
- add_kubernetes_metadata:
    default_indexers.enabled: false
    default_matchers.enabled: false

logging.level: info

output.kafka:
   enabled: true
   hosts: ["1.0.1.0:9092"]
   topics:
   - topic: "logs-hz"
     when.or:
       - contains:
           kubernetes.kubernetes.namespace: "ac-test"
       - contains:
           kubernetes.kubernetes.namespace: "iot-test"
   - topic: "nginx"
     when:
       contains:
         kubernetes.labels.app: "ingress-nginx"
   parttion.round_robin:
      reacheble_only: false
   required_acks: 1
   commpression: gzip
   max_message_bytes: 1000000

output.elasticsearch:
  enabled: false
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

One more thing, is the syntax below correct?

   topics:
   - topic: "logs-hz"
     when.or:
       - equals:
           kubernetes.namespace: "ac-test"
       - equals:
           kubernetes.namespace: "masa-iot-test"
---
   topics:
   - topic: "logs_backend_dev"
       when:
         and:
         - equals:
             type: "backend"
         - equals:
             environment: "dev"
7

Can you revise the document to add detailed instructions?

8

The condition looks good, but does the event really contain the field kubernetes.kubernetes.namespace? I would say that this field should be kubernetes.namespace.

This syntax looks mostly correct, but I think that in the second snippet, when should be at the same indentation level as topic, like this:

   topics:
   - topic: "logs_backend_dev"
     when:
       and:
         ...
9

ok, i made some low-level mistakes, thanks for the correction.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.