Unable to send data to AWS MSK, getting debug message: Kafka publish failed with: circuit breaker is open

saroja · February 11, 2021, 6:32pm

Dear Team ,

I am facing issue during data transfer in between filebeat and AWS msk cluster. The only error its showing like "circuit breaker open". I am using filebeat agent version 7.10.0 and AWS msk version 2.2.1. I am using TLS encryption. I have configured it in filebeat.yml file.. Filebeat harvestor starts reading the file and tring to establish the connection. Just after few seconds, Circuiy breaker open error is displaying.

Please advice if anyone is facing similar issue.

Please find the filebeat config here :-
#-------------------------------- Kafka Output --------------------------------
output.kafka:

enabled: true

hosts: [ "b-2.-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-3.-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-1.***-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094" ]

topic: "topic2"
worker: 3

Use TLS settings for encryption

tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]

Please find the Debug logs

2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z "@ "@ "beat": "filebeat",
"type": "_doc",
"version": "7.10.0"
}.... DEBUG [reload] cfgfile/list.go:63 Starting reload procedure, current runners: 0
DEBUG [harvester] log/harvester.go:583 Setting offset for file based on seek: /var/tmp/dummy_test7.log
DEBUG [reload] cfgfile/list.go:81 Start list: 0, Stop list: 0
DEBUG [harvester] log/harvester.go:569 Setting offset for file: /var/tmp/dummy_test7.log. Offset: 0
DEBUG [harvester] log/harvester.go:644 newLogFileReader with config.MaxBytes: 10485760
DEBUG [harvester] log/harvester.go:207 Harvester setup successful. Line terminator: 1
INFO cfgfile/reload.go:224 Loading of config files completed.
DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::71-64772 PrevId: Finished:false Fileinfo:0xc0007f69c0 Source:/var/tmp/dummy_test7.log Offset:0 Timestamp:2021-02-11 15:30:36.500252386 +0000 UTC m=+0.100246631 TTL:-1ns Type:log Meta:map FileStateOS:71-64772 IdentifierName:native} TimeSeries:false}
DEBUG [harvester] log/harvester.go:488 Update state: /var/tmp/dummy_test7.log, offset: 0
DEBUG [input] log/input.go:226 input states cleaned up. Before: 2, After: 2, Pending: 0
DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 2 active states.
DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::73-64772 PrevId: Finished:true Fileinfo: Source:/var/tmp/dummy_test5.log Offset:0 Timestamp:2021-02-11 15:30:36.499246667 +0000 UTC m=+0.099240912 TTL:0s Type:log Meta:map FileStateOS:73-64772 IdentifierName:native} TimeSeries:false}
DEBUG [input] log/input.go:238 Remove state for file as file removed: /var/tmp/dummy_test5.log
DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
INFO log/harvester.go:302 Harvester started for file: /var/tmp/dummy_test7.log
DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
DEBUG [state] file/states.go:134 State removed for /var/tmp/dummy_test5.log because of older: 0s
DEBUG [registrar] registrar/registrar.go:254 Registrar states cleaned up. Before: 2, After: 1, Pending: 0
DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 1 active states.
DEBUG [processors] processing/processors.go:203 Publish event: {
timestamp": "2021-02-11T15:30:36.501Z",
metadata": {

2021-02-11T15:30:36.514Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.

2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:37.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:39.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:40.472Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.473Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:223 done

warkolm · February 14, 2021, 11:28pm

Please don't create multiple topics on the same question, it makes it harder for people to help. Let's continue the discussion here Unable to send data to AWS MSK, getting debug message: Kafka publish failed with: circuit breaker is open.