Filebeat kafka output hash.hash get negative partition cause stuck

[Vvv (Victor)]

Summarry

I use filebeat to collect logs and output to kafka. Due to partition.hash.hash config hash message then mod to select partition, in some situation, this may cause filebeat stuck, e.g.:

massage: 2304669687
hash value: 2147483648
partition nums: 3
partitoin: -2

Environment

• Version: filebeat-7.6.2
• OS: CentOS Linux 7 (Core)

Steps to Reproduce

Kafka

Create kafka topic hashtest with 3 partitions.

Filebeat

log file

mock.log, I simply put in filebeat directory.

2304669686
2304669687

filebeat.yml

#================================= Nginx Input ================================
filebeat.inputs:
- type: log
  paths:
    - ./mock.log

#================================= Kafka Output ================================
output.kafka:
  enabled: true
  # replace with kafka cluster bootstrap servers
  hosts: ["%kafka cluster%"]
  topic: "hashtest"
  partition.hash:
    hash: ["message"]

run

./filebeat -e -d "*"

Output

Kafka

Only 1 message, and the 2nd message stuck, didn't pushlish to kafka.

[dev@sh-tjpm-emr-hermeskafka-qa-01 kafka_2.11-1.0.1]$ ./bin/kafka-console-consumer.sh --bootstrap-server 10.11.32.56:9092,10.11.33.223:9092,10.11.34.187:9092 --topic hashtest
{"@timestamp":"2021-02-23T05:20:42.857Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.6.2"},"log":{"offset":0,"file":{"path":"/data/filebeat/mock.log"}},"message":"2304669686","input":{"type":"log"},"ecs":{"version":"1.4.0"},"host":{"name":"sh-tjpm-emr-hermestest-qa-01"},"agent":{"ephemeral_id":"cdfbc223-fb5b-42c5-8842-ab79fd33add2","hostname":"sh-tjpm-emr-hermestest-qa-01","id":"b7a4f5aa-4ec8-4381-8f32-b74d949d92fe","version":"7.6.2","type":"filebeat"}}

Filebeat

endless loop

2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:169	got event.Meta["partition"] = -2
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:179	got event.Meta["topic"] = hashtest
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:273	finished kafka batch
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:287	Kafka publish failed with: kafka: partitioner returned an invalid partition index
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:169	got event.Meta["partition"] = -2

Analyze

Process

1. hash(2304669687) = 2147483648
2. Int32(2147483648) = -2147483648
3. -(-2147483648) = -2147483648
4. -2147483648 % 3 = -2
5. Kafka doesn't have -2 partition, STUCK.

Kafka doesn't have -2 partition, so it stuck forever.

Core

The absolute value Int32 MIN_VALUE equivalent to itself, still a negative number.

Pictures

Output

Kafka

Filebeat

Local Debugger