Filebeat find error Dropping event: no topic could be selected

Elastic Stack Beats
1

filebeat Helm chart deploy ，autodiscover kubernetes log . find ERROR: kafka/client.go:131 Dropping event: no topic could be selected. cause CPU 300% /var/lib/docker/containers/ dir path of filebeat docker log Disk usage has been growing rapidly . Please help to solve the problem ，thanks.

filebeat docker log disk

sh-4.2# du -sh *
0       checkpoints
8.0K    config.v2.json
1.0G    f4a9ff18206508e0427200ef36bbab107cad6df332960ff037709b7476fdf1ce-json.log
4.7G    f4a9ff18206508e0427200ef36bbab107cad6df332960ff037709b7476fdf1ce-json.log.1
4.7G    f4a9ff18206508e0427200ef36bbab107cad6df332960ff037709b7476fdf1ce-json.log.2
4.7G    f4a9ff18206508e0427200ef36bbab107cad6df332960ff037709b7476fdf1ce-json.log.3
4.7G    f4a9ff18206508e0427200ef36bbab107cad6df332960ff037709b7476fdf1ce-json.log.4
4.0K    hostconfig.json
0       mounts
sh-4.2# pwd
    /var/lib/docker/containers/f4a9ff18206508e0427200ef36bbab107cad6df332960ff037709b7476fdf1ce

    sh-4.2# tail -n 10 f4a9ff18206508e0427200ef36bbab107cad6df332960ff037709b7476fdf1ce-json.log|more
    {"log":"2019-05-07T04:02:47.699Z\u0009ERROR\u0009kafka/client.go:131\u0009Dropping event: no topic could be selected\n","
    stream":"stderr","time":"2019-05-07T04:02:47.699384057Z"}
    {"log":"2019-05-07T04:02:47.699Z\u0009ERROR\u0009kafka/client.go:131\u0009Dropping event: no topic could be selected\n","
    stream":"stderr","time":"2019-05-07T04:02:47.699386084Z"}
    {"log":"2019-05-07T04:02:47.699Z\u0009ERROR\u0009kafka/client.go:131\u0009Dropping event: no topic could be selected\n","
    stream":"stderr","time":"2019-05-07T04:02:47.699409054Z"}
    {"log":"2019-05-07T04:02:47.699Z\u0009ERROR\u0009kafka/client.go:131\u0009Dropping event: no topic could be selected\n","
    stream":"stderr","time":"2019-05-07T04:02:47.6994131Z"}
    {"log":"2019-05-07T04:02:47.699Z\u0009ERROR\u0009kafka/client.go:131\u0009Dropping event: no topic could be selected\n","
    stream":"stderr","time":"2019-05-07T04:02:47.699415269Z"}
    {"log":"2019-05-07T04:02:47.699Z\u0009ERROR\u0009kafka/client.go:131\u0009Dropping event: no topic could be selected\n","
    stream":"stderr","time":"2019-05-07T04:02:47.699418641Z"}

filebeat CPU %

Tasks:   3 total,   1 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.6 us,  4.8 sy,  0.0 ni, 86.5 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 13202631+total, 79643392 free, 18388488 used, 33994436 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 11127580+avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                             
    1 root      20   0 3781368  88624  20332 S 332.9  0.1 580:06.64 filebeat

filebeat.yml

fields:
  kubernetes.cluster.name: testcluster
fields_under_root: true
filebeat.autodiscover:
  providers:
  - hints.enabled: true
    templates:
    - condition.and:
      - not.has_fields:
        - kubernetes.labels.log-index
      - not.equals.kubernetes.container.name: filebeat
      config:
      - containers.ids:
        - ${data.kubernetes.container.id}
        exclude_lines:
        - ^\s+[\-`('.|_]
        fields:
          log_topic: testcluster-${data.kubernetes.namespace}
        type: docker
    type: kubernetes
filebeat.config:
  inputs:
    path: ${path.config}/inputs.d/*.yml
    reload.enabled: false
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
http.enabled: true
http.port: 5066
output.kafka:
  compression: gzip
  hosts:
  - kafka.testclustercom:31090
  - kafka.testcluster.com:31091
  - kafka.testclustercom:31092
  max_message_bytes: 1000000
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  topic: '%{[fields.log_topic]}'
processors:
- add_cloud_metadata: null
2

there was a reported case Filebeat Autodiscover Hints Breaking Template
when allowing hints started emitting of logs from all the containers even system ones. So I guess if these are emitted and log_topic is generated only for the ones matching condition then Kafka topic fallback to empty for others and therefore the error message. This is my guess.

I guess you cannot identify the source of the message. maybe you can play a little with hints.enabled to see if it helps

3

Hello, @Michal_Pristas Thank you for the help
closing hints. Enabled: true ，back to normal . Filebeat Autodiscover Hints Breaking Template ，This case is didn't use templates . Excuse me， hints. Enabled: true and templates Whether can be used at the same time，thanks.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.