Filebeat certificate error with kafka output + replicats

Elastic Stack Beats
1

Hi

I have a first problem with SSL usage with the output to kafka.
It seems that filebeat truncates the fqdn, in the configuration I use "kafka1.domain.fr" and it tells me that the certification is invalid for "kafka1".
I tried to delete the DNS query and he tells me that the IP is not valid. Which is normal because my certificate is a wildcard linked to a hostname and not an ip
This problem only occurs with filebeat and not with logstash

My configuration is like this

	output.kafka:
	   # initial brokers for reading cluster metadata
	   hosts: ["kafka1.domaine.fr:9092","kafka2.domaine.fr:9092"]

	   # message topic selection + partitioning
	   topic: WEB-TEST_APACHE
	   partition.round_robin:
	   reachable_only: false

	   required_acks: 1
	   compression: gzip
	   max_message_bytes: 1000000

Version Filebeat : 5.5.2-x86_64.rpm
OS : Centos 7.3

Here is the error log

2017-09-11T11:29:29+02:00 WARN client/metadata fetching metadata for [WEB-TEST_APACHE] from broker  :9093
2017-09-11T11:29:29+02:00 WARN producer/broker/0 starting up
2017-09-11T11:29:29+02:00 WARN producer/broker/0 state change to [open] on WEB-TEST_APACHE/0
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 selected broker 0
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 state change to [flushing-12]
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 state change to [normal]
2017-09-11T11:29:29+02:00 WARN Failed to connect to broker kafka1:9093: x509: certificate is valid for *.domaine.fr, domaine.fr, not kafka1
2017-09-11T11:29:29+02:00 WARN producer/broker/0 state change to [closing] because x509: certificate is valid for *.domaine.fr, domaine.fr, not kafka1
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 state change to [retrying-13]
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 abandoning broker 0
2017-09-11T11:29:29+02:00 WARN producer/broker/0 shut down


2017-09-11T11:29:29+02:00 WARN client/metadata fetching metadata for [WEB-TEST_APACHE] from broker kafka2.domaine.fr:9093
2017-09-11T11:29:29+02:00 WARN producer/broker/0 starting up
2017-09-11T11:29:29+02:00 WARN producer/broker/0 state change to [open] on WEB-TEST_APACHE/0
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 selected broker 0
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 state change to [flushing-13]
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 state change to [normal]
2017-09-11T11:29:29+02:00 WARN Failed to connect to broker kafka1:9093: x509: certificate is valid for *.domaine.fr, domaine.fr, not kafka1
2017-09-11T11:29:29+02:00 WARN producer/broker/0 state change to [closing] because x509: certificate is valid for *.domaine.fr, domaine.fr, not kafka1
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 state change to [retrying-14]
2017-09-11T11:29:29+02:00 WARN producer/leader/WEB-TEST_APACHE/0 abandoning broker 0
2017-09-11T11:29:29+02:00 WARN producer/broker/0 shut down

I have a second question about replication this time.
On Kafka we use the option "-replication-factor 2 " when creating subjects to replicate on the different nodes.
But with filebeat, topics are created automatically. In this case how can we give the information to replicate itself?

Thank you in advance
Kind regards
William

2

It seems that filebeat truncates the fqdn, in the configuration I use "kafka1.domain.fr" and it tells me that the certification is invalid for "kafka1"

It's not filebeat truncating names. It's your kafka setup. The kafka brokers do exchange the 'advertised_name' when joining a cluster. This name is passed to clients on connection bootstrap.

Btw. The reachable_only: false setting is missing 2 spaces.

Replication is handled by the kafka brokers.

3

Hi

Indeed it works perfectly !

I put in kafka server.properties this :

advertised.listeners=PLAINTEXT://kafka1.domaine.fr:9092,SSL://kafka1.domaine.fr:9093
advertised.host.name=kafka1.domaine.fr

Thank you very much

ps : I'm looking for replication, I will edit this post when I found the solution

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.