Unable to receive the log from filebeat to kafka

Elastic Stack Beats
1

Hi,

We are using kafka 2.12 and configure the output as below in the filebeat.

filebeat.prospectors:

  • type: log

    enabled: false

    paths:

    • /var/log/*.log

#============================= Filebeat modules ===============================

filebeat.config.modules:

Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#-------------------------- Kafka Output ----------------------------------
output.kafka:

initial brokers for reading cluster metadata

hosts: ["localhost:9092"]

message topic selection + partitioning

topic: 'logs'
partition.round_robin:
reachable_only: false

required_acks: 1
compression: gzip
max_message_bytes: 1000000

logging:
files:
rotateeverybytes: 10485760 # = 10MB

We have check the logs in the kafka there is no log on it.

2

Could you share the Filebeat version you are using and the filebeat log file? Please make sure to format it (same for the config above).

3

filebeat-6.1.1

Filebeat log given below :

INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=3313200 beat.memstats.memory_total=3313200 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=0

4

Not trying to hijack the thread, but i too have the exact issue.
Created a new thread "Unable send logs from filebeat 6 to kafka"
I have tried with my old filebeat 5.x version and its working perfect.

5

Can you both share your full log (best in debug mode) and config?