Logstash Kafka Input backpressure from Persistent Queue

[Suryatej_Mukkamalla]

This is my current ELK setup:

Logstash (7.2.0):
Input: Kafka
Queue: Persistent Queue (16 GB)
Output: Elasticsearch

Elasticsearch (7.2.0)

I am currently load testing Elasticsearch (trying to insert a billion documents 1KB in size into a single index in Elasticsearch) and I wanted some clarification on persistent queue behavior in Logstash.

According to the persistent queue documentation, if the persistent queue is full, then Logstash reduces the rate of input data flow. It also states that each input handles back pressure independently. My question is, how does the Kafka input plugin handle back pressure? Will it simply stop consuming events until there is space available in the queue?

[Badger]

@Suryatej_Mukkamalla

Will it simply stop consuming events until there is space available in the queue?

I cannot think of any alternative. It cannot accept an event from kafka (resulting in kafka committing the delivery) if it cannot persist the event in the queue. It is inconceivable to me that it would hold it memory and risk non-delivery.

That said, I haven't tested it.

[Suryatej_Mukkamalla]

Yes, that was my thought as well. I'll run a simple test with a small persistent queue (1 GB) and saturate it to see if any events are lost.