Hi,
I deployed filebeat on a host and seeing below errors:
Filebeat container log:
2018/10/22 11:22:26.464563 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:22:26.464592 logger.go:22: INFO done
2018/10/22 11:22:27.464719 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018/10/22 11:22:27.464894 logger.go:22: INFO retryer: send unwait-signal to consumer
2018/10/22 11:22:27.464929 logger.go:22: INFO done
2018/10/22 11:22:27.464944 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:22:27.464953 logger.go:22: INFO done
2018/10/22 11:22:27.466360 client.go:651: INFO Connected to Elasticsearch version 5.6.2
2018/10/22 11:22:27.468340 load.go:73: INFO Template already exists and will not be overwritten.
2018/10/22 11:22:27.468449 logger.go:22: INFO retryer: send unwait-signal to consumer
2018/10/22 11:22:27.468470 logger.go:22: INFO done
Filebeat debug log:
2018/10/22 11:21:37.902411 client.go:469: DBG Bulk item insert failed (i=37, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902432 client.go:469: DBG Bulk item insert failed (i=38, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902449 client.go:469: DBG Bulk item insert failed (i=39, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902571 client.go:469: DBG Bulk item insert failed (i=44, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902590 client.go:469: DBG Bulk item insert failed (i=45, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902608 client.go:469: DBG Bulk item insert failed (i=46, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902625 client.go:469: DBG Bulk item insert failed (i=47, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902644 client.go:469: DBG Bulk item insert failed (i=48, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902660 client.go:469: DBG Bulk item insert failed (i=49, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:38.066584 log.go:85: DBG End of file reached: /usr/share/filebeat/logs/zuul/log/zuul-debug.log; Backoff now.
2018/10/22 11:21:38.902914 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018/10/22 11:21:38.902988 client.go:627: DBG ES Ping(url=http://xxx:9200)
2018/10/22 11:21:38.903086 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:21:38.903105 logger.go:22: INFO done
2018/10/22 11:21:38.908598 client.go:650: DBG Ping status code: 200
2018/10/22 11:21:38.908635 client.go:651: INFO Connected to Elasticsearch version 5.6.2
2018/10/22 11:21:38.908806 client.go:669: DBG HEAD http://xxxt:9200/_template/filebeat <nil>
2018/10/22 11:21:38.914028 load.go:73: INFO Template already exists and will not be overwritten.
2018/10/22 11:21:38.914176 logger.go:22: INFO retryer: send unwait-signal to consumer
Filebeat config:
filebeat.prospectors:
- type: log
enabled: true
paths:
- /path/to/log
encoding: plain
include_lines: ['\d{4}\-\d{2}\-\d{2}']
tail_files: true
multiline.pattern: '^\d{4}\-\d{2}\-\d{2}'
multiline.negate: false
multiline.match: after
fields:
document_type: some_log
clean_removed: true
registry: /usr/share/filebeat/data/registry
setup.template:
name: "filebeat"
pattern: "filebeat-*"
settings:
index.number_of_shards: 1
index.number_of_replicas: 1
output.elasticsearch:
hosts: ["XXX:9200"]
index: "filebeat-%{[fields.document_type]}-%{+yyyy.MM.dd}"
Elasticsearch log:
Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.
Content type detection for rest requests is deprecated. Specify the
How to setup 'content type' without adding the whole new template manually (Because i want to use default filebeat template)? And i also i don't want to set it up manually for every index!