ProducerConfig values not all set

Elastic Stack Logstash
1

Hello, i've been using the kafka plugin and set some ssl settings in the plugin. The config checker (before starting logstash) detects my config values, but the producerConfig (in the log) is not set.

my config:

input {
  file {
      path => ["/some/path/here/*.txt"]
  }
}

output {
  kafka{
    topic_id => "ssl-txt"
    bootstrap_servers => "192.168.0.131:9092"
    ssl_key_password => "notRealPassword"
    ssl_keystore_location => "/some/path/here/KeyStore.jks"
    ssl_keystore_password => "notRealPassword"
    ssl_truststore_location => "/some/path/here/truststore.jks"
    ssl_truststore_password => "notRealPassword"
    ssl_endpoint_identification_algorithm => ""
  }
}

The log:

[2022-06-29T16:06:48,316][INFO ][org.apache.kafka.clients.producer.ProducerConfig][main] ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [192.168.0.131:9092]
buffer.memory = 33554432
client.dns.lookup = default
client.id = producer-1
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes =
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metadata.max.idle.ms = 300000
metric.reporters =
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 50
reconnect.backoff.ms = 50
request.timeout.ms = 40000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

[2022-06-29T16:06:48,855][INFO ][org.apache.kafka.common.utils.AppInfoParser][main] Kafka version: 2.5.1
[2022-06-29T16:06:48,856][INFO ][org.apache.kafka.common.utils.AppInfoParser][main] Kafka commitId: 0efa8fb0f4c73d92
[2022-06-29T16:06:48,856][INFO ][org.apache.kafka.common.utils.AppInfoParser][main] Kafka startTimeMs: 1656493608795
[2022-06-29T16:06:49,147][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/root/logstash-8.1.2/config/log-kafka-ssl.conf"], :thread=>"#<Thread:0x56c9dbe2 run>"}
[2022-06-29T16:06:49,657][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:50,273][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:50,835][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:51,451][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:52,045][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:52,627][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:52,628][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>3.48}
[2022-06-29T16:06:52,729][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/root/logstash-8.1.2/data/plugins/inputs/file/.sincedb_d14d6a239098033359cf73ee366f4382", :path=>["/root/log/sample/*.txt"]}
[2022-06-29T16:06:53,213][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:53,219][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-29T16:06:53,823][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:53,834][INFO ][filewatch.observingtail ][main][17a5648ac88418d3dd5ccea4a9ab81485c2d252c98a637cce7069edf2536e054] START, creating Discoverer, Watch with file and sincedb collections
[2022-06-29T16:06:53,842][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2022-06-29T16:06:54,437][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:54,976][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:55,509][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected
[2022-06-29T16:06:56,122][WARN ][org.apache.kafka.clients.NetworkClient][main] [Producer clientId=producer-1] Bootstrap broker 192.168.0.131:9092 (id: -1 rack: null) disconnected

2

Found the problem, need to add this in the plugin:
security_protocol => "SSL"

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.