# Failed to publish events: unavailable\_shards\_exception

**URL:** <https://discuss.elastic.co/t/failed-to-publish-events-unavailable-shards-exception/153393>\
**Category:** Elasticsearch\
**Created:** [October 22, 2018, 11:53am UTC](https://discuss.elastic.co/t/failed-to-publish-events-unavailable-shards-exception/153393 "2018-10-22T11:53:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![elk11](https://avatars.discourse-cdn.com/v4/letter/e/45deac/32.png) [@elk11](https://discuss.elastic.co/u/elk11)\
**Post date:** [October 22, 2018, 11:53am UTC](https://discuss.elastic.co/t/failed-to-publish-events-unavailable-shards-exception/153393/1 "2018-10-22T11:53:17Z")

</div>

Hi,

I deployed filebeat on a host and seeing below errors:

Filebeat container log:

```
2018/10/22 11:22:26.464563 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:22:26.464592 logger.go:22: INFO done
2018/10/22 11:22:27.464719 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018/10/22 11:22:27.464894 logger.go:22: INFO retryer: send unwait-signal to consumer
2018/10/22 11:22:27.464929 logger.go:22: INFO done
2018/10/22 11:22:27.464944 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:22:27.464953 logger.go:22: INFO done
2018/10/22 11:22:27.466360 client.go:651: INFO Connected to Elasticsearch version 5.6.2
2018/10/22 11:22:27.468340 load.go:73: INFO Template already exists and will not be overwritten.
2018/10/22 11:22:27.468449 logger.go:22: INFO retryer: send unwait-signal to consumer
2018/10/22 11:22:27.468470 logger.go:22: INFO done

```

Filebeat debug log:

```
2018/10/22 11:21:37.902411 client.go:469: DBG Bulk item insert failed (i=37, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902432 client.go:469: DBG Bulk item insert failed (i=38, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902449 client.go:469: DBG Bulk item insert failed (i=39, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902571 client.go:469: DBG Bulk item insert failed (i=44, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902590 client.go:469: DBG Bulk item insert failed (i=45, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902608 client.go:469: DBG Bulk item insert failed (i=46, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902625 client.go:469: DBG Bulk item insert failed (i=47, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902644 client.go:469: DBG Bulk item insert failed (i=48, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:37.902660 client.go:469: DBG Bulk item insert failed (i=49, status=503): {"type":"unavailable_shards_exception","reason":"[filebeat-zuul_log-2018.10.22][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[filebeat-zuul_log-2018.10.22][0]] containing [50] requests]"}
2018/10/22 11:21:38.066584 log.go:85: DBG End of file reached: /usr/share/filebeat/logs/zuul/log/zuul-debug.log; Backoff now.
2018/10/22 11:21:38.902914 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018/10/22 11:21:38.902988 client.go:627: DBG ES Ping(url=http://xxx:9200)
2018/10/22 11:21:38.903086 logger.go:22: INFO retryer: send wait signal to consumer
2018/10/22 11:21:38.903105 logger.go:22: INFO done
2018/10/22 11:21:38.908598 client.go:650: DBG Ping status code: 200
2018/10/22 11:21:38.908635 client.go:651: INFO Connected to Elasticsearch version 5.6.2
2018/10/22 11:21:38.908806 client.go:669: DBG HEAD http://xxxt:9200/_template/filebeat <nil>
2018/10/22 11:21:38.914028 load.go:73: INFO Template already exists and will not be overwritten.
2018/10/22 11:21:38.914176 logger.go:22: INFO retryer: send unwait-signal to consumer

```

Filebeat config:

```
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /path/to/log
  encoding: plain
  include_lines: ['\d{4}\-\d{2}\-\d{2}']
  tail_files: true
  multiline.pattern: '^\d{4}\-\d{2}\-\d{2}'
  multiline.negate: false
  multiline.match: after
  fields:
    document_type: some_log
  clean_removed: true
  registry: /usr/share/filebeat/data/registry

setup.template:
  name: "filebeat"
  pattern: "filebeat-*"
  settings:
    index.number_of_shards: 1
    index.number_of_replicas: 1

output.elasticsearch:
  hosts: ["XXX:9200"]
  index: "filebeat-%{[fields.document_type]}-%{+yyyy.MM.dd}"

```

Elasticsearch log:

```
Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.
Content type detection for rest requests is deprecated. Specify the 

```

How to setup 'content type' without adding the whole new template manually (Because i want to use default filebeat template)? And i also i don't want to set it up manually for every index!

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 22, 2018, 3:34pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-unavailable-shards-exception/153393/2 "2018-10-22T15:34:31Z")

</div>

Looks like you are using a very old version of filebeat.

---

<div class="post-metadata">

**Author:** ![elk11](https://avatars.discourse-cdn.com/v4/letter/e/45deac/32.png) [@elk11](https://discuss.elastic.co/u/elk11)\
**Post date:** [October 22, 2018, 4:50pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-unavailable-shards-exception/153393/3 "2018-10-22T16:50:08Z")

</div>

I'm using filebeat 6.0.1

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 23, 2018, 10:08am UTC](https://discuss.elastic.co/t/failed-to-publish-events-unavailable-shards-exception/153393/4 "2018-10-23T10:08:36Z")

</div>

The Elasticsearch error message does not seem to be triggered by filebeat. Filebeat 6.x sends the Contents-Type header.

The `unavailable_shards_exception` seems to be the problem. Have you checked your cluster is stable?

Moving to Elasticsearch Forum.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 20, 2018, 10:08am UTC](https://discuss.elastic.co/t/failed-to-publish-events-unavailable-shards-exception/153393/5 "2018-11-20T10:08:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.