Logstash Kafka Output Plugin loss of events or delay in delivering events


I have a problem sending events to cluster kafka.
The events read in input from a file are not always received on the kafka cluster (
it would seem only the first after the logstash process started).
From the logstash logs I can see that the file is correctly detected, after that I don't see any log towards the kafka queue.
I assume this is a logstash configuration problem, related to the configuration of the kafka cluster.

This is my logstash configuration file (Logstash version 7.1.1):

input {

file {
path => "/dati/LOG_ASCC_DISP/ASCC_DL_*"
start_position => beginning
sincedb_path => "/dev/null"
#type => "csv"
codec => multiline {
negate => true
pattern => "^[0-9]{8}|"
what => "previous"
output {

kafka {
codec => json
topic_id => "in.dispositivi"
acks => "1"
client_id => "XXXXXXXXXXXX"
bootstrap_servers => "ICDCOLCMA:9092"
security_protocol => "SSL"
#ssl_truststore_location => "/home/logstash/logstash-7.1.1/config/certificati/logstash.server.truststore.jks"
#ssl_truststore_password => "xxxxxxxxxxx"
ssl_truststore_location => "/dati/logstash/logstash-7.1.1/keystore.jks"
ssl_truststore_password => "ICDCOLCMA"
#ssl_keystore_location => "/home/logstash/logstash-7.1.1/config/certificati/logstash.server.keystore.jks"
ssl_keystore_location => "/dati/logstash/logstash-7.1.1/keystore.jks"
ssl_keystore_password => "ICDCOLCMA"
ssl_endpoint_identification_algorithm => ""

the kafka side configuration (kafka version 2.12-2.1.1)

it has only one consumer which in turn sees other 3 followers (one of the 3 followers is the consumer itself). each topic has 3 partitions with relative replicas.

The same process logstash tested on a single kafka node (non-clustered) works perfectly.

can someone help me?


As a sanity check, can you publish messages to the topic using the console producer?

Thanks for the suggestion.
Sending the event through the console it is received correctly.
The problem is related to the ruby ​​code of the logstash process configuration file.
Some additions made to the code inhibited the sending of the event.
Excluding that code events are sent regularly.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.