Dear Team ,
I am facing issue during data transfer in between filebeat and AWS msk cluster. The only error its showing like "circuit breaker open". I am using filebeat agent version 7.10.0 and AWS msk version 2.2.1. I am using TLS encryption. I have configured it in filebeat.yml file.. Filebeat harvestor starts reading the file and tring to establish the connection. Just after few seconds, Circuiy breaker open error is displaying.
Please advice if anyone is facing similar issue.
Please find the filebeat config here :-
#-------------------------------- Kafka Output --------------------------------
output.kafka:
enabled: true
hosts: [ "b-2.-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-3.-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-1.***-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094" ]
topic: "topic2"
worker: 3
Use TLS settings for encryption
tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]
Please find the Debug logs
2021-02-11T15:30:36.500Z DEBUG [reload] cfgfile/list.go:63 Starting reload procedure, current runners: 0
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:583 Setting offset for file based on seek: /var/tmp/dummy_test7.log
2021-02-11T15:30:36.500Z DEBUG [reload] cfgfile/list.go:81 Start list: 0, Stop list: 0
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:569 Setting offset for file: /var/tmp/dummy_test7.log. Offset: 0
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:644 newLogFileReader with config.MaxBytes: 10485760
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:207 Harvester setup successful. Line terminator: 1
2021-02-11T15:30:36.500Z INFO cfgfile/reload.go:224 Loading of config files completed.
2021-02-11T15:30:36.500Z DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2021-02-11T15:30:36.500Z DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::71-64772 PrevId: Finished:false Fileinfo:0xc0007f69c0 Source:/var/tmp/dummy_test7.log Offset:0 Timestamp:2021-02-11 15:30:36.500252386 +0000 UTC m=+0.100246631 TTL:-1ns Type:log Meta:map FileStateOS:71-64772 IdentifierName:native} TimeSeries:false}
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:488 Update state: /var/tmp/dummy_test7.log, offset: 0
2021-02-11T15:30:36.500Z DEBUG [input] log/input.go:226 input states cleaned up. Before: 2, After: 2, Pending: 0
2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 2 active states.
2021-02-11T15:30:36.500Z DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2021-02-11T15:30:36.501Z DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::73-64772 PrevId: Finished:true Fileinfo: Source:/var/tmp/dummy_test5.log Offset:0 Timestamp:2021-02-11 15:30:36.499246667 +0000 UTC m=+0.099240912 TTL:0s Type:log Meta:map FileStateOS:73-64772 IdentifierName:native} TimeSeries:false}
2021-02-11T15:30:36.501Z DEBUG [input] log/input.go:238 Remove state for file as file removed: /var/tmp/dummy_test5.log
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2021-02-11T15:30:36.501Z INFO log/harvester.go:302 Harvester started for file: /var/tmp/dummy_test7.log
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2021-02-11T15:30:36.501Z DEBUG [state] file/states.go:134 State removed for /var/tmp/dummy_test5.log because of older: 0s
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:254 Registrar states cleaned up. Before: 2, After: 1, Pending: 0
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 1 active states.
2021-02-11T15:30:36.501Z DEBUG [processors] processing/processors.go:203 Publish event: {
"@timestamp": "2021-02-11T15:30:36.501Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.10.0"
}....
2021-02-11T15:30:36.514Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:37.501Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to kafka(b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)
2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:37.501Z DEBUG [kafka] kafka/client.go:96 connect: [b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094]
2021-02-11T15:30:37.501Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to kafka(b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) established
2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:37.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:39.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:40.472Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.473Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:223 done