Failed to publish events: unavailable_shards_exception

Elastic Stack Elasticsearch
1

Hi,

I deployed filebeat on a host and seeing below errors:

Filebeat container log:

2018/10/22 11:22:26.464563 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:22:26.464592 logger.go:22: INFO   done
2018/10/22 11:22:27.464719 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018/10/22 11:22:27.464894 logger.go:22: INFO retryer: send unwait-signal to consumer
2018/10/22 11:22:27.464929 logger.go:22: INFO   done
2018/10/22 11:22:27.464944 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:22:27.464953 logger.go:22: INFO   done
2018/10/22 11:22:27.466360 client.go:651: INFO Connected to Elasticsearch version 5.6.2
2018/10/22 11:22:27.468340 load.go:73: INFO Template already exists and will not be overwritten.
2018/10/22 11:22:27.468449 logger.go:22: INFO retryer: send unwait-signal to consumer
2018/10/22 11:22:27.468470 logger.go:22: INFO   done

Filebeat debug log:

2018/10/22 11:21:37.902411 client.go:469: DBG Bulk item insert failed (i=37, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902432 client.go:469: DBG Bulk item insert failed (i=38, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902449 client.go:469: DBG Bulk item insert failed (i=39, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902571 client.go:469: DBG Bulk item insert failed (i=44, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902590 client.go:469: DBG Bulk item insert failed (i=45, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902608 client.go:469: DBG Bulk item insert failed (i=46, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902625 client.go:469: DBG Bulk item insert failed (i=47, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902644 client.go:469: DBG Bulk item insert failed (i=48, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902660 client.go:469: DBG Bulk item insert failed (i=49, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:38.066584 log.go:85: DBG End of file reached: /usr/share/filebeat/logs/zuul/log/zuul-debug.log; Backoff now.
2018/10/22 11:21:38.902914 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018/10/22 11:21:38.902988 client.go:627: DBG ES Ping(url=http://xxx:9200)
2018/10/22 11:21:38.903086 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:21:38.903105 logger.go:22: INFO   done
2018/10/22 11:21:38.908598 client.go:650: DBG Ping status code: 200
2018/10/22 11:21:38.908635 client.go:651: INFO Connected to Elasticsearch version 5.6.2
2018/10/22 11:21:38.908806 client.go:669: DBG HEAD http://xxxt:9200/_template/filebeat  <nil>
2018/10/22 11:21:38.914028 load.go:73: INFO Template already exists and will not be overwritten.
2018/10/22 11:21:38.914176 logger.go:22: INFO retryer: send unwait-signal to consumer

Filebeat config:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /path/to/log
  encoding: plain
  include_lines: ['\d{4}\-\d{2}\-\d{2}']
  tail_files: true
  multiline.pattern: '^\d{4}\-\d{2}\-\d{2}'
  multiline.negate: false
  multiline.match: after
  fields:
    document_type: some_log
  clean_removed: true
  registry: /usr/share/filebeat/data/registry

setup.template:
  name: "filebeat"
  pattern: "filebeat-*"
  settings:
    index.number_of_shards: 1
    index.number_of_replicas: 1

output.elasticsearch:
  hosts: ["XXX:9200"]
  index: "filebeat-%{[fields.document_type]}-%{+yyyy.MM.dd}"

Elasticsearch log:

Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.
Content type detection for rest requests is deprecated. Specify the

How to setup 'content type' without adding the whole new template manually (Because i want to use default filebeat template)? And i also i don't want to set it up manually for every index!

2

Looks like you are using a very old version of filebeat.

3

I'm using filebeat 6.0.1

4

The Elasticsearch error message does not seem to be triggered by filebeat. Filebeat 6.x sends the Contents-Type header.

The unavailable_shards_exception seems to be the problem. Have you checked your cluster is stable?

Moving to Elasticsearch Forum.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.