Filebeat do not send logs to kafka

Elastic Stack Beats
1

Hello Everyone,

We are trying implement a centralized logging solution filebeats (on eks) ->kafka -> elasticsearch(aws). I have tried different versions of filebeat 6.X and 7.X with different valid versions of kafka. File beat does not run because 'Readiness probe failed: kafka output doesn't support testing' I am using helm charts to install filebeat. When exec in to filebeat pod. the module list is empty. I added. Even than the
'Readiness probe fails and kafka is not enabled.

My question is what are module related config need to done under filebeat.yml:
in values.yml that s used by helm. Are not all modules by default loaded ( when installed via helm)?

Thank you for the help
Ram

      23     filebeat.config.modules:
      24        path: ${path.config}/modules.d/*.yml
      25     filebeat.modules:
     26       -  module: nginx
     27       -  module: kafka

Below is filebeatConfig: section of values.yml

4 filebeatConfig:
  5   filebeat.yml: |
  6     filebeat.inputs:
  7     - type: log
  8       paths:
  9         - /var/log/containers/*.log
 10       processors:
 11       - add_kubernetes_metadata:
 12           host: ${NODE_NAME}
 13           matchers:
 14           - logs_path:
 15               logs_path: "/var/log/containers/"
 16     output.kafka:
 17        enabled: true
 18        hosts: ["host1:9092","host2:9092"]
 19        topic: "Application_logs"
 20        codec.format:
 21           string: '%{[@timestamp]} %{[message]}'
 22
 23     filebeat.config.modules:
 24       path: ${path.config}/modules.d/*.yml
 25     filebeat.modules:
 26     - module: nginx
 27     - module: kafka ```
2

Did you verify the connectivity? Are your hosts reachable from this host?

3

Yes, connectivity is good. I actually tested with kafka cli, to send topic an message to kafka cluster. it works

4

Can anyone assure me that above output configuration is correct ?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.