# Negative Kafka partition bug

**URL:** <https://discuss.elastic.co/t/negative-kafka-partition-bug/270046>\
**Category:** Beats\
**Tags:** libbeat\
**Created:** [April 13, 2021, 6:00pm UTC](https://discuss.elastic.co/t/negative-kafka-partition-bug/270046 "2021-04-13T18:00:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cbrown184](https://avatars.discourse-cdn.com/v4/letter/c/779978/32.png) [@cbrown184](https://discuss.elastic.co/u/cbrown184)\
**Post date:** [April 13, 2021, 6:00pm UTC](https://discuss.elastic.co/t/negative-kafka-partition-bug/270046/1 "2021-04-13T18:00:51Z")

</div>

Hi - we use Filebeat to output our logs to Kafka. We got hit by a nasty bug in prod where Filebeat gets stuck in an endless loop.

It was previously documented on this thread [Filebeat kafka output hash.hash get negative partition cause stuck](https://discuss.elastic.co/t/filebeat-kafka-output-hash-hash-get-negative-partition-cause-stuck/265153)

I have proposed a low-impact fix here.

> <https://github.com/elastic/beats/pull/25048>
>
> \## What does this PR do?
> 
> Fix a bug in Kafka output partition to hash method which results in a negative Kafka partition.
> Demo of the bug available on my other branch. https://github.com/cbrown184/beats/tree/negative\_partition\_edge\_case\_demo\_test
> 
> You can reproduce this with any key that gives the hash value of 2147483648. E.g the string "16000002EFEBA11E"
> 
> \## Why is it important?
> 
> When the partition key hash is equal to 2147483648 Filebeat gets stuck in an endless loop.
> 
> \## Checklist
> 
> \<!-- Mandatory
> Add a checklist of things that are required to be reviewed in order to have the PR approved
> 
> List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~ )
> \--\>
> 
> \- \[x\] My code follows the style guidelines of this project
> \- \[\] ~~I have commented my code, particularly in hard-to-understand areas~~
> \- \[\] ~~I have made corresponding changes to the documentation~~
> \- \[\] ~~I have made corresponding change to the default configuration files~~
> \- \[x\] I have added tests that prove my fix is effective or that my feature works
> \- \[x\] I have added an entry in \`CHANGELOG.next.asciidoc\` or \`CHANGELOG-developer.next.asciidoc\`.
> 
> \## Author's Checklist
> 
> \<!-- Recommended
> Add a checklist of things that are required to be reviewed in order to have the PR approved
> \--\>
> \- \[\]
> 
> \## How to test this PR locally
> 
> \## Related issues
> 
> This bug was discussed previously in this thread.
> https://discuss.elastic.co/t/filebeat-kafka-output-hash-hash-get-negative-partition-cause-stuck/265153
> 
> \## Use cases
> 
> hash(16000002EFEBA11E) = 2147483648
> Int32(2147483648) = -2147483648
> \-(-2147483648) = -2147483648
> \-2147483648 % 3 = -2
> Kafka doesn't have -2 partition, STUCK.
> Kafka doesn't have -2 partition, so the kafka producer becomes stuck forever.
> 
> \## Screenshots
> 
> \## Logs
> 
> Filebeat endless loop
> 
> 2021-02-23T13:18:40.911+0800	DEBUG	\[kafka\]	kafka/client.go:169	got event.Meta\["partition"\] = -2
> 2021-02-23T13:18:40.911+0800	DEBUG	\[kafka\]	kafka/client.go:179	got event.Meta\["topic"\] = hashtest
> 2021-02-23T13:18:40.911+0800	DEBUG	\[kafka\]	kafka/client.go:273	finished kafka batch
> 2021-02-23T13:18:40.911+0800	DEBUG	\[kafka\]	kafka/client.go:287	Kafka publish failed with: kafka: partitioner returned an invalid partition index
> 2021-02-23T13:18:40.911+0800	DEBUG	\[kafka\]	kafka/client.go:169	got event.Me
> 
> 
> Error Logs in production
> 2021-04-09T13:48:02Z DBG Kafka publish failed with: kafka: partitioner returned an invalid partition index
> 
> Demo test failing
> === RUN TestHashMaxIntPlusOneDoesNotReturnNegativePartition
> partition\_test.go:328: 
> Error Trace:	partition\_test.go:328
> Error: Should be true
> Test: TestHashMaxIntPlusOneDoesNotReturnNegativePartition
> \--- FAIL: TestHashMaxIntPlusOneDoesNotReturnNegativePartition (0.00s)
> 
> FAIL
> 
> Expected :8
> Actual :-8

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [April 13, 2021, 11:38pm UTC](https://discuss.elastic.co/t/negative-kafka-partition-bug/270046/2 "2021-04-13T23:38:01Z")

</div>

Hey @cbrown184 ! Thanks for letting us know, I have added the correct labels to the PR so that the correct team will get a notification, will let you know how it goes in the PR itself most likely! 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 13, 2021, 12:45pm UTC](https://discuss.elastic.co/t/negative-kafka-partition-bug/270046/4 "2021-05-13T12:45:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.