Warning is observed huge number of times 'input V2 factory.CheckConfig failed to clone config before checking it. Original config will be checked, it might trigger an input duplication warning: failed to get 'id': missing field accessing 'id''

Elastic Stack Beats
1

There is one small clarification needed regarding the below warning.
Actually, when autodiscover is enabled and tried to deploy the service, we see the below warning repeated continuously with huge number of occurrences.
Warning:
input V2 factory.CheckConfig failed to clone config before checking it. Original config will be checked, it might trigger an input duplication warning: failed to get 'id': missing field accessing 'id'
Configuration used:

filebeat.inputs:
    - type: log
      paths:
      - /var/lib/docker/containers/**/*.log
      - /var/log/pods/**/*.log
      fields:
        logplane: "app-logs"
      fields_under_root: true
      tail_files: true
      close_timeout: "5m"
      id: "ls_id"
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        namespace: "testnamespace"
        hints.enabled: true
        add_resource_metadata:
          deployment: false
          cronjob: false
          namespace:
            enabled: false
          node:
            enabled: false
        include_annotations:
        hints.default_config:
          type: container
          paths:
            - /var/log/pods/${data.kubernetes.namespace}_${data.kubernetes.pod.name}_${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/*.log
        templates:
        appenders:
          - type: config
            config:
              fields:
                logplane: "app-logs"
              fields_under_root: true
              close_timeout: "5m"
              processors:
              - decode_json_fields:
                  fields: ["message"]
                  target: "json"
                  max_depth: 1
              - drop_event:
                  when:
                    or:
                      - not.or:
                          - equals:
                              kubernetes.labels.app: "loggen"
              - drop_fields:
                  fields:
                    - "kubernetes.annotations"
                  ignore_missing: true
    output.logstash:
      hosts: "logstash:5044"
      ssl.certificate_authorities: "${TRUSTED_INTERNAL_ROOT_CA_PATH}/ca.crt"
      ssl.certificate: "${LT_CLIENT_CERT_PATH}/${CERT}"
      ssl.key: "${LT_CLIENT_CERT_PATH}/${KEY}"
      ssl.verification_mode: "full"
      ssl.renegotiation: "freely"
      ssl.supported_protocols: ["TLSv1.2", "TLSv1.3"]
      ssl.cipher_suites: []
      bulk_max_size: 2048
      worker: 1
      pipelining: 0
      ttl: 30
      queue.mem:
        flush.timeout: 1s
    filebeat.registry.flush: 5s
    logging.level: "info"
    logging.metrics.enabled: false
    http.enabled: true
    http.host: localhost
    http.port: 5066

We just enabled autodiscover and deployed the service, then we observe huge number of warnings even though we did not add any filestream input.
Actually this warning is observed from filebeat v8.16.1 and this is the github issue filebeat: input v2 compat uses random ID for CheckConfig by AndersonQ · Pull Request #41585 · elastic/beats · GitHub. In this commit, generateCheckConfig() is added and also the warning is added which we suspect this warning might be due to this added code.
Can you please let us know in which scenario this warning will be observed and what is the expected behavior and also how to avoid this warning to proceed further.

Thanks & regards,
Sunanda N.

2

Hi @Sunanda, thanks for reporting this!

Indeed there is a bug on emitting this log warning, it should be emitted only for the Filestream input, but it is being emitted to every input when autodiscover is used.

It is harmless and the input is started as expected. You can ignore it or add an ID to suppress it.

I also created a GitHub issue to track it: Missing input ID warning is emitted for all inputs when using autodiscover · Issue #42969 · elastic/beats · GitHub

1 Like