I have created AWS MSK service with SASL_SSL authentication and able to send/receive data through python code to it. i am trying to send metricbeat and getting the below error. this if from windows machine
Here is the metricbeat.yaml file configuration:
'''
output.kafka:
enabled: true
hosts: ["b-1.xxxxxxxxxxxxxxxxxxxx.c2.kafka.us-east-1.amazonaws.com:9096","b-2.xxxxxxxxxxxxxxxxxxxx.c2.kafka.us-east-1.amazonaws.com:9096"]
topic: "testtopic"
username: 'username'
password: 'password'
sasl.mechanism: 'SCRAM-SHA-512'
partition.round_robin:
reachable_only: false
codec.json:
pretty: true
escape_html: true
required_acks: 1
compression: gzip
max_message_bytes: 1000000
'''
Error message:
2023-09-26T11:48:32.956Z INFO instance/beat.go:647 Home path: [C:\Application\metricbeat-7.8.0-windows-x86_64] Config path: [C:\Application\metricbeat-7.8.0-windows-x86_64] Data path: [C:\Application\metricbeat-7.8.0-windows-x86_64\data] Logs path: [C:\Application\metricbeat-7.8.0-windows-x86_64\logs]
2023-09-26T11:48:32.966Z INFO instance/beat.go:655 Beat ID: 49985462-308d-4641-81c9-8320f30e3e2c
2023-09-26T11:48:32.975Z INFO [beat] instance/beat.go:983 Beat info {"system_info": {"beat": {"path": {"config": "C:\Application\metricbeat-7.8.0-windows-x86_64", "data": "C:\Application\metricbeat-7.8.0-windows-x86_64\data", "home": "C:\Application\metricbeat-7.8.0-windows-x86_64", "logs": "C:\Application\metricbeat-7.8.0-windows-x86_64\logs"}, "type": "metricbeat", "uuid": "49985462-308d-4641-81c9-8320f30e3e2c"}}}
2023-09-26T11:48:32.977Z INFO [beat] instance/beat.go:992 Build info {"system_info": {"build": {"commit": "f79387d32717d79f689d94fda1ec80b2cf285d30", "libbeat": "7.8.0", "time": "2020-06-14T18:20:02.000Z", "version": "7.8.0"}}}
2023-09-26T11:48:32.977Z INFO [beat] instance/beat.go:995 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.13.10"}}}
2023-09-26T11:48:32.981Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"536720785143"},"availability_zone":"us-east-1a","image":{"id":"ami-0fd3967a5af8fa3fb"},"instance":{"id":"i-096c576abbbee75fc"},"machine":{"type":"m4.large"},"provider":"aws","region":"us-east-1"}
2023-09-26T11:48:32.984Z INFO [beat] instance/beat.go:999 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-09-26T05:51:19.11Z","name":"EC2AMAZ-PSJ1K33","ip":["fe80::9c0:a701:70b7:fac4/64","100.110.25.203/27","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.4737 (WinBuild.160101.0800)","mac":["0e:3f:a6:4f:b2:63"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2019 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.4737"},"timezone":"GMT","timezone_offset_sec":0,"id":"2ec4b320-d4d3-4a60-833b-851ba42bb69d"}}}
2023-09-26T11:48:32.984Z INFO [beat] instance/beat.go:1028 Process info {"system_info": {"process": {"cwd": "C:\Application\metricbeat-7.8.0-windows-x86_64", "exe": "C:\Application\metricbeat-7.8.0-windows-x86_64\metricbeat.exe", "name": "metricbeat.exe", "pid": 5980, "ppid": 512, "start_time": "2023-09-26T11:48:32.114Z"}}}
2023-09-26T11:48:32.984Z INFO instance/beat.go:310 Setup Beat: metricbeat; Version: 7.8.0
2023-09-26T11:48:32.985Z INFO [publisher] pipeline/module.go:113 Beat name: EC2AMAZ-PSJ1K33
2023-09-26T11:48:33.013Z INFO instance/beat.go:463 metricbeat start running.
2023-09-26T11:48:33.018Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2023-09-26T11:48:33.020Z INFO helper/privileges_windows.go:79 Metricbeat process and system info: {"OSVersion":{"Major":6,"Minor":2,"Build":9200},"Arch":"amd64","NumCPU":2,"User":{"SID":"S-1-5-21-739377628-2922313706-3570492944-500","Account":"Administrator","Domain":"EC2AMAZ-PSJ1K33","Type":1},"ProcessPrivs":{"SeBackupPrivilege":{"enabled":false},"SeChangeNotifyPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateGlobalPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePagefilePrivilege":{"enabled":false},"SeCreateSymbolicLinkPrivilege":{"enabled":false},"SeDebugPrivilege":{"enabled":true},"SeDelegateSessionUserImpersonatePrivilege":{"enabled":false},"SeImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseBasePriorityPrivilege":{"enabled":false},"SeIncreaseQuotaPrivilege":{"enabled":false},"SeIncreaseWorkingSetPrivilege":{"enabled":false},"SeLoadDriverPrivilege":{"enabled":false},"SeManageVolumePrivilege":{"enabled":false},"SeProfileSingleProcessPrivilege":{"enabled":false},"SeRemoteShutdownPrivilege":{"enabled":false},"SeRestorePrivilege":{"enabled":false},"SeSecurityPrivilege":{"enabled":false},"SeShutdownPrivilege":{"enabled":false},"SeSystemEnvironmentPrivilege":{"enabled":false},"SeSystemProfilePrivilege":{"enabled":false},"SeSystemtimePrivilege":{"enabled":false},"SeTakeOwnershipPrivilege":{"enabled":false},"SeTimeZonePrivilege":{"enabled":false},"SeUndockPrivilege":{"enabled":false}}}
2023-09-26T11:48:33.020Z INFO helper/privileges_windows.go:87 SeDebugPrivilege is enabled. SeDebugPrivilege=(Enabled)
2023-09-26T11:48:33.021Z INFO cfgfile/reload.go:164 Config reloader started
2023-09-26T11:48:33.023Z INFO cfgfile/reload.go:224 Loading of config files completed.
2023-09-26T11:48:34.035Z INFO [publisher_pipeline_output] pipeline/output.go:144 Connecting to kafka(b-1.xxxxxxxxxxxxxx.c2.kafka.us-east-1.amazonaws.com:9096,b-2.xxxxxxxxxxxxxx.c2.kafka.us-east-1.amazonaws.com:9096)
2023-09-26T11:48:34.035Z INFO [publisher_pipeline_output] pipeline/output.go:152 Connection to kafka(b-1.xxxxxxxxxxxxxxxxxxxxx.c2.kafka.us-east-1.amazonaws.com:9096) established
23> -09-26T11:48:34.036Z INFO [publisher] pipeline/retry.go:221 **retryer: send unwait signal to consumer
2023-09-26T11:48:34.036Z INFO [publisher] pipeline/retry.go:225 done
2023-09-26T11:48:36.427Z INFO [publisher] pipeline/retry.go:161 Drop batch
2023-09-26T11:48:41.129Z INFO [publisher] pipeline/retry.go:161 Drop batch
2023-09-26T11:48:44.026Z INFO [publisher] pipeline/retry.go:161 Drop batch
Getting the below error when debug mode enabled..
kafka publish failed with : circuit breaker is open error
Please help me to resolve this issue.