Unable to receive the log from filebeat to kafka


We are using kafka 2.12 and configure the output as below in the filebeat.


  • type: log

    enabled: false


    • /var/log/*.log

#============================= Filebeat modules ===============================


Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: false

#==================== Elasticsearch template setting ==========================

index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#-------------------------- Kafka Output ----------------------------------

initial brokers for reading cluster metadata

hosts: ["localhost:9092"]

message topic selection + partitioning

topic: 'logs'
reachable_only: false

required_acks: 1
compression: gzip
max_message_bytes: 1000000

rotateeverybytes: 10485760 # = 10MB

We have check the logs in the kafka there is no log on it.

Could you share the Filebeat version you are using and the filebeat log file? Please make sure to format it (same for the config above).


Filebeat log given below :

INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=3313200 beat.memstats.memory_total=3313200 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=0

Not trying to hijack the thread, but i too have the exact issue.
Created a new thread "Unable send logs from filebeat 6 to kafka"
I have tried with my old filebeat 5.x version and its working perfect.

Can you both share your full log (best in debug mode) and config?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.