I am trying to set up ssl connection between filebeat 5.2 and kafka 1.0 using the below steps. I am fairly new to encryption world and seeing errors during this process. Not sure if I am missing something in cert/key or filebeat and kafka config. Would appreciate some guidance.
Filebeat: v5.2 residing on Debian Jessie 8
Kafka: v1.0 residing on Debian Wheezy 7
Sequence followed
Login to common node
cat > cert_info_filebeat << EOF
[req]
default_bits = 2048
prompt = no
default_md = sha512
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=US
ST=<xxx>
L=<xxx>
O=<xxx>
OU=<xxx>
emailAddress=<xxx@xx.com>
CN = filebeat
[ req_ext ]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[ alt_names ]
IP.1 = <filebeat IP 1>
IP.2 = <filebeat IP 2>
IP.3 = <filebeat IP 3>
[ usr_cert ]
# Extensions for server certificates.
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "OpenSSL FileBeat Server / Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation
extendedKeyUsage = serverAuth, clientAuth
EOF
Run these commands
Generates key
openssl genrsa -out filebeat.key 2048
Generates certificate signing request
openssl req -new -out filebeat.csr -key filebeat.key -config <(cat cert_info_filebeat)
Generate self signed certificate
openssl x509 -req -days 3650 -in filebeat.csr -signkey filebeat.key -out filebeat.crt -extensions req_ext -extfile cert_info_filebeat
After this I have filebeat.key, filebeat.csr, filebeat.crt files. Ran this command to verify crt and no errors there
openssl x509 -infilebeat.crt -text -noout
Modify filebeat config
output.kafka:
hosts: ["<kafka IP>:9093"]
topic: '%{[type]}'
# ssl crt/key files
ssl.certificate: "/etc/filebeat_softlayer.crt"
ssl.key: "/etc/filebeat_softlayer.key"
# ssl.verification_mode: none ( turn off ssl domain/hostname verifications )
compression: gzip
Kafka Server
cat > cert_info_kafka << EOF
[req]
default_bits = 2048
prompt = no
default_md = sha512
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=US
ST=<xxx>
L=<xxx>
O=<xxx>
OU=<xxx>
emailAddress=<xxx>
CN = kafka
[ req_ext ]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[ alt_names ]
IP.1 = <kafka IP>
[ usr_cert ]
# Extensions for server certificates.
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "OpenSSL Kafka Server / Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation
extendedKeyUsage = serverAuth, clientAuth
EOF
Generate key
openssl genrsa -out kafka.key 2048
Generate csr
openssl req -new -out kafka.csr -key kafka.key -config <(cat cert_info_kafka)
openssl req -text -noout -in kafka.csr
Generate cert
openssl x509 -req -days 3650 -in kafka.csr -signkey kafka.key -out kafka.crt -extensions req_ext -extfile cert_info_kafka
Generate pkcs12 file so we can import the crt and key to generate kafka keystore file
openssl pkcs12 -export -in kafka.crt -inkey kafka.key -out kafka.p12 -name kafka-pk12
cd /opt/obuildfactory/jdk-1.8.0-openjdk-x86_64/jre/bin/
./keytool -importkeystore -deststorepass <password> -destkeystore kafka-keystore.jks -srckeystore kafka_softlayer.p12 -srcstoretype PKCS12
Modify kafka config
# Listener List - Comma-separated list of URIs we will listen on and the listener names.
# If the listener name is not a security protocol, listener.security.protocol.map must also be set.
# Specify hostname as 0.0.0.0 to bind to all interfaces. Leave hostname empty to bind to default interface.
# Examples of legal listener lists: PLAINTEXT://myhost:9092,SSL://:9091 CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093
#listeners=PLAINTEXT://172.16.0.110:9092
listeners=ONSL://:9092,INSL://:9093
# advertised listener part
advertised.listeners=ONSL://<kafka IP>:9092,INSL://<kafka IP>:9093
inter.broker.listener.name=INSL
listener.security.protocol.map=INSL:SSL,ONSL:PLAINTEXT
# ssl encryption config from docs
ssl.keystore.location=/opt/obuildfactory/jdk-1.8.0-openjdk-x86_64/jre/bin/kafka-keystore.jks
ssl.keystore.password=<password>
Issues:
When I start kafka process on kafka node the logs says but process and ports are still up and running
[2018-02-08 15:09:16,060] ERROR [Controller id=1, targetBrokerId=1] Connection to node 1 failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
Filebeat error
2018-02-07T20:35:04Z WARN client/metadata fetching metadata for all topics from broker <kafka IP>:9093
2018-02-07T20:35:04Z WARN Failed to connect to broker <kafka IP>:9093: x509: certificate signed by unknown authority
2018-02-07T20:35:04Z WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA x509.UnknownAuthorityError=x509: certificate signed by unknown authority)
2018-02-07T20:35:04Z WARN kafka message: client/metadata no available broker to send metadata request to
2018-02-07T20:35:04Z WARN client/brokers resurrecting 1 dead seed brokers
2018-02-07T20:35:04Z WARN client/metadata retrying after 250ms... (3 attempts remaining)
2018-02-07T20:35:05Z WARN client/metadata fetching metadata for all topics from broker <kafka IP>:9093
When I run this command from filebeat node
openssl s_client -connect :9093 -showcerts
It shows this message
verify error:num=21:unable to verify the first certificate
verify return:1
Please advise on what I am missing and what are the steps to enable ssl between filebeat and kafka in local environment.
Thanks,
Gangadhar