Dear Team,
I am using filebeat agent 7.10.0 and AWS msk in my pipeline.
I am getting error : Kafka publish failed with: circuit breaker is open and retryer: send unwait signal to consumer.
Please suggest if anyone facing same type of issue and used specific configuration for the same
I am using below configuration for TLS encryption here
output.kafka:
enabled: true
hosts: [ "b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094",
"b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094" ]
topic: "shippinglog"
key: "Lckey"
worker: 3
Use TLS settings for encryption
tls.certificate_authorities: ["/tmp/kafka.client.truststore.jks"]
Debug messages -
Please find the Debug logs
2021-02-11T15:30:36.500Z DEBUG [reload] cfgfile/list.go:63 Starting reload procedure, current runners: 0
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:583 Setting offset for file based on seek: /var/tmp/dummy_test7.log
2021-02-11T15:30:36.500Z DEBUG [reload] cfgfile/list.go:81 Start list: 0, Stop list: 0
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:569 Setting offset for file: /var/tmp/dummy_test7.log. Offset: 0
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:644 newLogFileReader with config.MaxBytes: 10485760
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:207 Harvester setup successful. Line terminator: 1
2021-02-11T15:30:36.500Z INFO cfgfile/reload.go:224 Loading of config files completed.
2021-02-11T15:30:36.500Z DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2021-02-11T15:30:36.500Z DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::71-64772 PrevId: Finished:false Fileinfo:0xc0007f69c0 Source:/var/tmp/dummy_test7.log Offset:0 Timestamp:2021-02-11 15:30:36.500252386 +0000 UTC m=+0.100246631 TTL:-1ns Type:log Meta:map FileStateOS:71-64772 IdentifierName:native} TimeSeries:false}
2021-02-11T15:30:36.500Z DEBUG [harvester] log/harvester.go:488 Update state: /var/tmp/dummy_test7.log, offset: 0
2021-02-11T15:30:36.500Z DEBUG [input] log/input.go:226 input states cleaned up. Before: 2, After: 2, Pending: 0
2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2021-02-11T15:30:36.500Z DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 2 active states.
2021-02-11T15:30:36.500Z DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2021-02-11T15:30:36.501Z DEBUG [publisher] pipeline/client.go:231 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::73-64772 PrevId: Finished:true Fileinfo: Source:/var/tmp/dummy_test5.log Offset:0 Timestamp:2021-02-11 15:30:36.499246667 +0000 UTC m=+0.099240912 TTL:0s Type:log Meta:map FileStateOS:73-64772 IdentifierName:native} TimeSeries:false}
2021-02-11T15:30:36.501Z DEBUG [input] log/input.go:238 Remove state for file as file removed: /var/tmp/dummy_test5.log
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2021-02-11T15:30:36.501Z INFO log/harvester.go:302 Harvester started for file: /var/tmp/dummy_test7.log
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2021-02-11T15:30:36.501Z DEBUG [state] file/states.go:134 State removed for /var/tmp/dummy_test5.log because of older: 0s
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:254 Registrar states cleaned up. Before: 2, After: 1, Pending: 0
2021-02-11T15:30:36.501Z DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 1 active states.
2021-02-11T15:30:36.501Z DEBUG [processors] processing/processors.go:203 Publish event: {
"@timestamp": "2021-02-11T15:30:36.501Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.10.0"
}....
2021-02-11T15:30:36.514Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:37.501Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to kafka(b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094)
2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:37.501Z DEBUG [kafka] kafka/client.go:96 connect: [b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094 b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094]
2021-02-11T15:30:37.501Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to kafka(b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-2.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-3.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094,b-1.saroj-test-development.gv3pju.c6.kafka.eu-west-1.amazonaws.com:9094) established
2021-02-11T15:30:37.501Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:37.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:39.515Z DEBUG [harvester] log/log.go:107 End of file reached: /var/tmp/dummy_test7.log; Backoff now.
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.470Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.470Z INFO [publisher] pipeline/retry.go:223 done
2021-02-11T15:30:40.472Z DEBUG [kafka] kafka/client.go:277 finished kafka batch
2021-02-11T15:30:40.473Z DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-02-11T15:30:40.473Z INFO [publisher] pipeline/retry.go:223 done