filebeat: filebeat-7.10.2-linux-x86_64
kafka: kafka_2.13-2.6.0/
here is my filebeat config:
filebeat.inputs:
- type: log
# Change to true to enable this input configuration.
enabled: true
fields_under_root: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /home/work/test/log/*.log
#- c:\programdata\elasticsearch\logs\*
json.keys_under_root: true
json.add_error_key: false
json.overwrote_keys: true
output.kafka:
enabled: true
hosts: ["xxx1:9093",
"xxx2:9093"]
topic: "test3"
partition.round_robin:
reachable_only: false
compression: none
max_message_bytes: 1000000
sasl.mechanism:
kerberos.enabled: true
kerberos.auth_type: keytab
kerberos.username: "yuenxu"
kerberos.keytab: "/home/work/keytab/yuenxu.keytab"
kerberos.service_name: "kafka"
kerberos.config_path: "/home/work/krb5/krb5.conf"
kerberos.realm: "LZ.DSCC.99.COM"
About the kafka with kerberos, I have test it with console producer/consumer. It's OK.
Here is the filebeat error message:
2021-09-17T19:31:19.916+0800 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-09-17T19:31:19.916+0800 INFO [publisher] pipeline/retry.go:223 done
2021-09-17T19:31:19.996+0800 DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-09-17T19:31:19.996+0800 DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-09-17T19:31:19.996+0800 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-09-17T19:31:19.996+0800 INFO [publisher] pipeline/retry.go:223 done
Is there anything wrong? Thanks.