Hi,
We are sending cassandra logs to ES via filebeat and after each few days it stop sending entries to ES with following Error is visible in filebeat logs:
2022-02-09T16:04:08.762Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(https://<ES Host Name>:443)) established
2022-02-09T16:04:15.424Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
2022-02-09T16:04:15.424Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(https://<ES Host Name>:443))
2022-02-09T16:04:15.436Z INFO elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 6.0.0
2022-02-09T16:04:15.460Z INFO template/load.go:128 Template already exists and will not be overwritten.
2022-02-09T16:04:15.460Z INFO instance/beat.go:889 Template successfully loaded.
Restarting filebeat solves the issue and again after few days issue re-appears.
I have read through some of the posts in the forum with similar error messages but unable to link it to my issue.
filebeat version 6.8.12 (amd64), libbeat 6.8.12 [fdb5036adbe45aa10a03882b2245578ad17c3615 built 2020-08-12 06:26:46 +0000 UTC]
filebeat.yml
filebeat.prospectors:
- input_type: log
fields:
index: 54olqeye17
paths:
- "/var/log/cassandra/system.log*"
- "/var/log/cassandra/gc.log.*.current"
scan_frequency: 30s
document_type: cassandra_system_logs
exclude_files: ['\.zip$']
multiline.pattern: '^TRACE|DEBUG|WARN|INFO|ERROR'
multiline.negate: true
multiline.match: after
multiline.timeout: 5m
backoff: 5s
max_backoff: 10s
- input_type: log
fields:
index: 9q2beq2iuu
paths:
- "/var/log/cassandra/repair.log*"
document_type: cassandra_logs
exclude_files: ['\.zip$']
multiline.pattern: '^TRACE|DEBUG|WARN|INFO|ERROR'
multiline.negate: true
multiline.match: after
multiline.timeout: 5m
backoff: 5s
max_backoff: 10s
- input_type: log
fields:
index: 04wbin96l3
paths:
- "/var/log/cassandra/debug.log*"
document_type: cassandra_logs
exclude_files: ['\.zip$']
multiline.pattern: '^TRACE|DEBUG|WARN|INFO|ERROR'
multiline.negate: true
multiline.match: after
multiline.timeout: 5m
backoff: 5s
max_backoff: 10s
output.elasticsearch:
hosts: ["https://<ES Host Name>:443"]
index: '%{[fields.index]}'
ssl.certificate: "/var/private/es-client.pem"
ssl.key: "/var/private/es-client.key"
backoff.init: 5
setup.template:
name: anx
pattern: anx