How to set up kafka output partition option: hash.hash or key?

Elastic Stack Beats
1

Hello,
Can I set up kafka partitions as follows?
partition: hash
hash.hash to be combination of beat.hostname + source?
key to be the same as beat.hostname + source?

I am wondering what should be the correct syntax for the above setting? I could not find an example.

thanks!
yan

2

There is a difference between key based and non-key based hashing. As you want to hash based on hostname and source, you want it to be not-key based (as key will become the kafka key, overwriting recent events).

e.g.

output.kafka:
  partition.hash:
    hash: ['source', 'beat.hostname']
    random: true # if false non-hashable events will be dropped
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.