Kafka input plugin

[Samuel_Delepiere]

The documentation does not really make sense when it comes to poll_timeout_ms and fetch_max_wait_ms.

According to the documentation:

• fetch_max_wait_ms has a default value of 500ms
• poll_timeout_ms has a default value of 100ms.

Yet it says that fetch_max_wait_ms should be less than or equal to poll_timeout_ms.

Additionally setting poll_timeout_ms does not seem to do anything. I can't see a difference in the properties of the kafka consumer when changing this value.

Could you provide some explanations please?

Thanks.

[Badger]

I cannot see any reason for fetch_max_wait_ms to be constrained to be smaller than poll_timeout_ms.

poll_timeout_ms is not a property of the kafka consumer itself, it is used in the calling code. It determines how long each call to consumer.poll waits to get data. The default value is set in the plugin.

fetch_max_wait_ms is a property of the consumer and the default value is set by kafka. This SO question explicitly covers the case where fetch_max_wait_ms is larger than poll_timeout_ms. There will be multiple poll calls until fetch_max_wait_ms expires and then the server will return whatever it has even if fetch_min_bytes has not been exceeded. Given that the default fetch_min_bytes is 1, it appears to me that the value of fetch_max_wait_ms does not usually matter.