Logstash kafka output - producer type for version 2.x

Elastic Stack Logstash
#1

Hi guys,

we need to configure sync producer type for kafka output plugin (or understand what is the default in version 2.x.)
there was such variable in version 1.5 but it was removed - https://www.elastic.co/guide/en/logstash/1.5/plugins-outputs-kafka.html#plugins-outputs-kafka-producer_type.

anyone has idea how to configure this in version 2.x?

thanks in advance,

Ronnysh

#2

The 0.9+ producer has the sync and async all built into one. You want acks to be 1 or all for a sync operation.

Keep in mind that Kafka has an internal buffer so you have to keep in mind https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html#plugins-outputs-kafka-block_on_buffer_full and https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html#plugins-outputs-kafka-buffer_memory all come into play.

Really if you need synchronized, known writes, you need to write your own Kafka output that can check the status of the message being submitted as right now Logstash doesn't do stuff like that so there is a small chance of losing some progress in the buffer if stuff crashed.

#3