# Filebeat kafka output hash.hash get negative partition cause stuck

**URL:** <https://discuss.elastic.co/t/filebeat-kafka-output-hash-hash-get-negative-partition-cause-stuck/265153>\
**Category:** Beats\
**Tags:** filebeat, libbeat\
**Created:** [February 23, 2021, 5:27am UTC](https://discuss.elastic.co/t/filebeat-kafka-output-hash-hash-get-negative-partition-cause-stuck/265153 "2021-02-23T05:27:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vvv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vvv/32/52878_2.png) [@Vvv](https://discuss.elastic.co/u/Vvv)\
**Post date:** [February 23, 2021, 5:27am UTC](https://discuss.elastic.co/t/filebeat-kafka-output-hash-hash-get-negative-partition-cause-stuck/265153/1 "2021-02-23T05:27:58Z")

</div>

## Summarry

I use filebeat to collect logs and output to kafka. Due to `partition.hash.hash` config `hash` message then `mod` to select partition, in some situation, this may cause filebeat stuck, e.g.:

```auto
massage: 2304669687
hash value: 2147483648
partition nums: 3
partitoin: -2

```

## Environment

- Version: filebeat-7.6.2
- OS: CentOS Linux 7 (Core)

## Steps to Reproduce

### Kafka

Create kafka topic `hashtest` with 3 partitions.

### Filebeat

#### log file

`mock.log`, I simply put in `filebeat` directory.

```auto
2304669686
2304669687

```

#### filebeat.yml

```auto
#================================= Nginx Input ================================
filebeat.inputs:
- type: log
  paths:
    - ./mock.log

#================================= Kafka Output ================================
output.kafka:
  enabled: true
  # replace with kafka cluster bootstrap servers
  hosts: ["%kafka cluster%"]
  topic: "hashtest"
  partition.hash:
    hash: ["message"]

```

#### run

```auto
./filebeat -e -d "*"

```

## Output

### Kafka

Only 1 message, and the 2nd message stuck, didn't pushlish to kafka.

```auto
[dev@sh-tjpm-emr-hermeskafka-qa-01 kafka_2.11-1.0.1]$ ./bin/kafka-console-consumer.sh --bootstrap-server 10.11.32.56:9092,10.11.33.223:9092,10.11.34.187:9092 --topic hashtest
{"@timestamp":"2021-02-23T05:20:42.857Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.6.2"},"log":{"offset":0,"file":{"path":"/data/filebeat/mock.log"}},"message":"2304669686","input":{"type":"log"},"ecs":{"version":"1.4.0"},"host":{"name":"sh-tjpm-emr-hermestest-qa-01"},"agent":{"ephemeral_id":"cdfbc223-fb5b-42c5-8842-ab79fd33add2","hostname":"sh-tjpm-emr-hermestest-qa-01","id":"b7a4f5aa-4ec8-4381-8f32-b74d949d92fe","version":"7.6.2","type":"filebeat"}}

```

### Filebeat

endless loop

```auto
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:169	got event.Meta["partition"] = -2
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:179	got event.Meta["topic"] = hashtest
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:273	finished kafka batch
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:287	Kafka publish failed with: kafka: partitioner returned an invalid partition index
2021-02-23T13:18:40.911+0800	DEBUG	[kafka]	kafka/client.go:169	got event.Meta["partition"] = -2

```

## Analyze

#### Process

1. hash(2304669687) = 2147483648
2. Int32(2147483648) = -2147483648
3. -(-2147483648) = -2147483648
4. -2147483648 % 3 = -2
5. Kafka doesn't have -2 partition, STUCK.

#### 

Kafka doesn't have -2 partition, so it stuck forever.

#### Core

The absolute value `Int32 MIN_VALUE` equivalent to itself, still a negative number.

## Pictures

### Output

#### Kafka

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/1/919a5bd72a308adf10868b45b3be0590b0d8f1de.png)

#### Filebeat

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/2/727e37b1ea2543f12ac2eb7998a153f502ea8fc9.png)

### Local Debugger

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/4/74aa3424197e62cd6aa91e0f06261d0e322bed3e.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2021, 7:28am UTC](https://discuss.elastic.co/t/filebeat-kafka-output-hash-hash-get-negative-partition-cause-stuck/265153/2 "2021-03-23T07:28:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.