Unable to receive the log from filebeat to kafka

(Saravana Ganesh) #1


We are using kafka 2.12 and configure the output as below in the filebeat.


  • type: log

    enabled: false


    • /var/log/*.log

#============================= Filebeat modules ===============================


Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: false

#==================== Elasticsearch template setting ==========================

index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#-------------------------- Kafka Output ----------------------------------

initial brokers for reading cluster metadata

hosts: ["localhost:9092"]

message topic selection + partitioning

topic: 'logs'
reachable_only: false

required_acks: 1
compression: gzip
max_message_bytes: 1000000

rotateeverybytes: 10485760 # = 10MB

We have check the logs in the kafka there is no log on it.

(ruflin) #2

Could you share the Filebeat version you are using and the filebeat log file? Please make sure to format it (same for the config above).

(Saravana Ganesh) #3


Filebeat log given below :

INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=3313200 beat.memstats.memory_total=3313200 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=0

(Kiran Tella) #4

Not trying to hijack the thread, but i too have the exact issue.
Created a new thread "Unable send logs from filebeat 6 to kafka"
I have tried with my old filebeat 5.x version and its working perfect.

(ruflin) #5

Can you both share your full log (best in debug mode) and config?

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.