Logstash Kafka Output Plugin loss of events or delay in delivering events

[giorgiogale (Giorgio Galeotti)]

Hi,

I have a problem sending events to cluster kafka.
The events read in input from a file are not always received on the kafka cluster (
it would seem only the first after the logstash process started).
From the logstash logs I can see that the file is correctly detected, after that I don't see any log towards the kafka queue.
I assume this is a logstash configuration problem, related to the configuration of the kafka cluster.

This is my logstash configuration file (Logstash version 7.1.1):

input {

file {
path => "/dati/LOG_ASCC_DISP/ASCC_DL_*"
start_position => beginning
sincedb_path => "/dev/null"
#type => "csv"
codec => multiline {
negate => true
pattern => "^[1]{8}|"
what => "previous"
}
.
.
.
.
.
.
output {

kafka {
codec => json
topic_id => "in.dispositivi"
acks => "1"
client_id => "XXXXXXXXXXXX"
bootstrap_servers => "ICDCOLCMA:9092"
security_protocol => "SSL"
#ssl_truststore_location => "/home/logstash/logstash-7.1.1/config/certificati/logstash.server.truststore.jks"
#ssl_truststore_password => "xxxxxxxxxxx"
ssl_truststore_location => "/dati/logstash/logstash-7.1.1/keystore.jks"
ssl_truststore_password => "ICDCOLCMA"
#ssl_keystore_location => "/home/logstash/logstash-7.1.1/config/certificati/logstash.server.keystore.jks"
ssl_keystore_location => "/dati/logstash/logstash-7.1.1/keystore.jks"
ssl_keystore_password => "ICDCOLCMA"
ssl_endpoint_identification_algorithm => ""

the kafka side configuration (kafka version 2.12-2.1.1)

it has only one consumer which in turn sees other 3 followers (one of the 3 followers is the consumer itself). each topic has 3 partitions with relative replicas.

The same process logstash tested on a single kafka node (non-clustered) works perfectly.

can someone help me?

Thanks

---

1. 0-9 ↩︎

[tobydickinson (Toby Dickinson Williams)]

As a sanity check, can you publish messages to the topic using the console producer?

[giorgiogale (Giorgio Galeotti)]

Thanks for the suggestion.
Sending the event through the console it is received correctly.
The problem is related to the ruby ​​code of the logstash process configuration file.
Some additions made to the code inhibited the sending of the event.
Excluding that code events are sent regularly.