Unable to send data to AWS MSK, getting error message: Kafka publish failed with: circuit breaker is open

saroja · February 11, 2021, 6:44pm

Dear Team,

I am using filebeat agent 7.10.0 and AWS msk in my pipeline.
I am getting error : Kafka publish failed with: circuit breaker is open and retryer: send unwait signal to consumer.

Please suggest if anyone facing same type of issue and used specific configuration for the same

I am using below configuration for TLS encryption here

output.kafka:

enabled: true

hosts: [ "b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094" ]

topic: "shippinglog"
key: "Lckey"
worker: 3

Use TLS settings for encryption

tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]

Debug messages -

Please find the Debug logs

2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.500Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z 2021-02-11T15:30:36.501Z "@ "@ "beat": "filebeat",
"type": "_doc",
"version": "7.10.0"
}.... DEBUG [reload] cfgfile/list.go:63 Starting reload procedure, current runners: 0
DEBUG [harvester] log/harvester.go:583 Setting offset for file based on seek: /var/tmp/dummy_test7.log
DEBUG [reload] cfgfile/list.go:81 Start list: 0, Stop list: 0
DEBUG [harvester] log/harvester.go:569 Setting offset for file: /var/tmp/dummy_test7.log. Offset: 0
DEBUG [harvester] log/harvester.go:644 newLogFileReader with config.MaxBytes: 10485760
DEBUG [harvester] log/harvester.go:207 Harvester setup successful. Line terminator: 1
INFO cfgfile/reload.go:224 Loading of config files completed.
DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::71-64772 PrevId: Finished:false Fileinfo:0xc0007f69c0 Source:/var/tmp/dummy_test7.log Offset:0 Timestamp:2021-02-11 15:30:36.500252386 +0000 UTC m=+0.100246631 TTL:-1ns Type:log Meta:map FileStateOS:71-64772 IdentifierName:native} TimeSeries:false}
DEBUG [harvester] log/harvester.go:488 Update state: /var/tmp/dummy_test7.log, offset: 0
DEBUG [input] log/input.go:226 input states cleaned up. Before: 2, After: 2, Pending: 0
DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 2 active states.
DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::73-64772 PrevId: Finished:true Fileinfo: Source:/var/tmp/dummy_test5.log Offset:0 Timestamp:2021-02-11 15:30:36.499246667 +0000 UTC m=+0.099240912 TTL:0s Type:log Meta:map FileStateOS:73-64772 IdentifierName:native} TimeSeries:false}
DEBUG [input] log/input.go:238 Remove state for file as file removed: /var/tmp/dummy_test5.log
DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
INFO log/harvester.go:302 Harvester started for file: /var/tmp/dummy_test7.log
DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
DEBUG [state] file/states.go:134 State removed for /var/tmp/dummy_test5.log because of older: 0s
DEBUG [registrar] registrar/registrar.go:254 Registrar states cleaned up. Before: 2, After: 1, Pending: 0
DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 1 active states.
DEBUG [processors] processing/processors.go:203 Publish event: {
timestamp": "2021-02-11T15:30:36.501Z",
metadata": {

2021-02-11T15:30:36.514Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.

2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:37.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:39.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:40.472Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.473Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:223 done

system · March 11, 2021, 8:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.