Can Filebeat send to one specific partition in Kafka cluster?

Elastic Stack Beats
1

Hello,

We have a scenario where we are sending data from one filbeat(6.3.2) agent to a three partition kafka(0.0.10.) topic. We are wanting filebeat to only send to one of the partitions in the kafka cluster.

I found this post where it kind of explains how to do this using the key feature in filebeat:

This is our filebeat config:

#----------------------------- Kafka output --------------------------------
output.kafka:

  # initial brokers for reading cluster metadata
  hosts: ["broker1.fmr.com:9092","broker2.fmr.com:9092","broker3.fmr.com:9092"]

  # message topic selection + partitioning
  topic: '%{[topic]}'
  client_id: my_clientid

  key: 'anodot_kafkatest'
  partition.hash:
    hash: []
    reachable_only: false

#  required_acks: 1
#  compression: gzip
#  max_message_bytes: 1000000

Does the config look correct? Should we be using something else?
Is there a way to verify that it in fact went to only one partition?

Thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.