# Unable to send data to AWS MSK, getting error message: Kafka publish failed with: circuit breaker is open **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 11, 2021, 6:44pm UTC](https://discuss.elastic.co/t/unable-to-send-data-to-aws-msk-getting-error-message-kafka-publish-failed-with-circuit-breaker-is-open/264045 "2021-02-11T18:44:51Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![saroja](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@saroja](https://discuss.elastic.co/u/saroja)\ **Post date:** [February 11, 2021, 6:44pm UTC](https://discuss.elastic.co/t/unable-to-send-data-to-aws-msk-getting-error-message-kafka-publish-failed-with-circuit-breaker-is-open/264045/1 "2021-02-11T18:44:51Z")
Dear Team, I am using filebeat agent 7.10.0 and AWS msk in my pipeline. I am getting error : Kafka publish failed with: circuit breaker is open and retryer: send unwait signal to consumer. Please suggest if anyone facing same type of issue and used specific configuration for the same I am using below configuration for TLS encryption here output.kafka: enabled: true hosts: ["[b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)", "[b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)", "[b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)" ] topic: "shippinglog" key: "Lckey" worker: 3 # Use TLS settings for encryption tls.certificate\_authorities: ["/tmp/kafka.client.truststore.jks"] Debug messages - ## Please find the Debug logs 2021-02-11T15:30:36.500Z DEBUG [reload] cfgfile/list.go:63 Starting reload procedure, current runners: 0 2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:583 Setting offset for file based on seek: /var/tmp/dummy\_test7.log 2021-02-11T15:30:36.500Z DEBUG [reload] cfgfile/list.go:81 Start list: 0, Stop list: 0 2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:569 Setting offset for file: /var/tmp/dummy\_test7.log. Offset: 0 2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:644 newLogFileReader with config.MaxBytes: 10485760 2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:207 Harvester setup successful. Line terminator: 1 2021-02-11T15:30:36.500Z INFO cfgfile/reload.go:224 Loading of config files completed. 2021-02-11T15:30:36.500Z DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1} 2021-02-11T15:30:36.500Z DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::71-64772 PrevId: Finished:false Fileinfo:0xc0007f69c0 Source:/var/tmp/dummy\_test7.log Offset:0 Timestamp:2021-02-11 15:30:36.500252386 +0000 UTC m=+0.100246631 TTL:-1ns Type:log Meta:map FileStateOS:71-64772 IdentifierName:native} TimeSeries:false} 2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:488 Update state: /var/tmp/dummy\_test7.log, offset: 0 2021-02-11T15:30:36.500Z DEBUG [input] log/input.go:226 input states cleaned up. Before: 2, After: 2, Pending: 0 2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:264 Processing 1 events 2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1 2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 2 active states. 2021-02-11T15:30:36.500Z DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1} 2021-02-11T15:30:36.501Z DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::73-64772 PrevId: Finished:true Fileinfo: Source:/var/tmp/dummy\_test5.log Offset:0 Timestamp:2021-02-11 15:30:36.499246667 +0000 UTC m=+0.099240912 TTL:0s Type:log Meta:map FileStateOS:73-64772 IdentifierName:native} TimeSeries:false} 2021-02-11T15:30:36.501Z DEBUG [input] log/input.go:238 Remove state for file as file removed: /var/tmp/dummy\_test5.log 2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:264 Processing 1 events 2021-02-11T15:30:36.501Z INFO log/harvester.go:302 Harvester started for file: /var/tmp/dummy\_test7.log 2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1 2021-02-11T15:30:36.501Z DEBUG [state] file/states.go:134 State removed for /var/tmp/dummy\_test5.log because of older: 0s 2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:254 Registrar states cleaned up. Before: 2, After: 1, Pending: 0 2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 1 active states. 2021-02-11T15:30:36.501Z DEBUG [processors] processing/processors.go:203 Publish event: { "@timestamp": "2021-02-11T15:30:36.501Z", "@metadata": { "beat": "filebeat", "type": "\_doc", "version": "7.10.0" }.... 2021-02-11T15:30:36.514Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy\_test7.log; Backoff now. 2021-02-11T15:30:37.501Z INFO [publisher\_pipeline\_output] pipeline/output.go:143 Connecting to kafka([b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)) 2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2021-02-11T15:30:37.501Z DEBUG [kafka] kafka/client.go:96 connect: [[b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) [b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)] 2021-02-11T15:30:37.501Z INFO [publisher\_pipeline\_output] pipeline/output.go:151 Connection to kafka([b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094),[b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094](http://b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)) established 2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:223 done 2021-02-11T15:30:37.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy\_test7.log; Backoff now. 2021-02-11T15:30:39.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy\_test7.log; Backoff now. 2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:277 finished kafka batch 2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open 2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:223 done 2021-02-11T15:30:40.472Z DEBUG [kafka] kafka/client.go:277 finished kafka batch 2021-02-11T15:30:40.473Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open 2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:223 done ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 11, 2021, 8:45pm UTC](https://discuss.elastic.co/t/unable-to-send-data-to-aws-msk-getting-error-message-kafka-publish-failed-with-circuit-breaker-is-open/264045/2 "2021-03-11T20:45:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.