# Failed to publish events: temporary bulk send failure

**URL:** <https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330>\
**Category:** Beats\
**Tags:** packetbeat\
**Created:** [June 3, 2018, 5:53pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330 "2018-06-03T17:53:32Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![nuwancs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nuwancs/32/46419_2.png) [@nuwancs](https://discuss.elastic.co/u/nuwancs)\
**Post date:** [June 3, 2018, 5:53pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/1 "2018-06-03T17:53:32Z")

</div>

I get the following error  
ERROR pipeline/output.go:92 c

when trying to add the following to the packetbeat.yml  
processors:

- include\_fields.fields: ["http.request.body"]

connection to elasticsearch is ok  
2018-06-03T23:10:37.835+0530 DEBUG [elasticsearch] elasticsearch/client.go:689 Ping status code: 200

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [June 4, 2018, 9:19am UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/2 "2018-06-04T09:19:08Z")

</div>

Could you share the full error you get or is that it? Please include the lines before and after. Could you also share your full config file and the version of packetbeat you use?

---

<div class="post-metadata">

**Author:** ![nuwancs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nuwancs/32/46419_2.png) [@nuwancs](https://discuss.elastic.co/u/nuwancs)\
**Post date:** [June 4, 2018, 2:03pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/3 "2018-06-04T14:03:48Z")

</div>

Root Cause as i found:  
when following is added to elasticsearch output in packetbeat.yml  
index: "packetbeat-%{[beat.version]}-%{+yyyy.MM.dd.HH}"

Environment:  
elasticsearch version - 6.2.4 packetbeat version - 6.2.4

Full Log:

```
2018-06-04T00:37:40.893+0530 ERROR pipeline/output.go:92 Failed to publish events: temporary bulk send failure
2018-06-04T00:37:40.893+0530 DEBUG [elasticsearch] elasticsearch/client.go:666 ES Ping(url=http://localhost:9200)
2018-06-04T00:37:40.894+0530 DEBUG [elasticsearch] elasticsearch/client.go:689 Ping status code: 200
2018-06-04T00:37:40.894+0530 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.2
2018-06-04T00:37:40.894+0530 DEBUG [elasticsearch] elasticsearch/client.go:708 HEAD http://localhost:9200/_template/packetbeat-6.2.4 <nil>
2018-06-04T00:37:40.895+0530 INFO template/load.go:73 Template already exists and will not be overwritten.
2018-06-04T00:37:40.896+0530 DEBUG [elasticsearch] elasticsearch/client.go:303 PublishEvents: 1 events have been published to elasticsearch in 1.245631ms.
2018-06-04T00:37:40.896+0530 DEBUG [elasticsearch] elasticsearch/client.go:507 Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}

```

Config:

```
#============================== Network device ================================
# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: any

#========================== Transaction protocols =============================
packetbeat.protocols:
- type: http
  # Configure the ports where to listen for HTTP traffic. You can disable
  # the HTTP protocol by commenting out the list of ports.
  ports: [80, 8080, 8000, 5000, 8002]
  send_all_headers: true	
  send_response: true
  send_request: true
  include_body_for: ["application/json"]

- type: tls
  # Configure the ports where to listen for TLS traffic. You can disable
  # the TLS protocol by commenting out the list of ports.
  ports: [443]

#==================== Elasticsearch template setting ==========================
# Set to false to disable template loading.
setup.template.enabled: true
# Overwrite existing template
setup.template.overwrite: true

setup.template.name: "packetbeat-%{[beat.version]}"
setup.template.pattern: "packetbeat-%{[beat.version]}-*"

setup.template.fields: "fields.yml"

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ Processors ===================================

processors:
  #- include_fields.fields: ["http.request.body"]
  - include_fields.fields: ["http.request.body","http.response.body","direction","http.request.headers.api_id","http.request.headers.api_name",
    "http.request.headers.api_publisher","http.request.headers.application_id","http.request.headers.application_name","http.request.headers.request_id",
    "http.request.headers.resource","http.request.headers.user_id","http.request.headers.version","http.response.code","http.response.phrase","ip"," path",
    "responsetime","status",'bytes_out','bytes_in','client_ip']

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "packetbeat-%{[beat.version]}-%{+yyyy.MM.dd.HH}"
```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [June 6, 2018, 11:53am UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/4 "2018-06-06T11:53:14Z")

</div>

I think the problem is that you remove the `beat.version` from the event with your include\_fields processor so the index is invalid. Add beat.version to your include list and try again.

---

<div class="post-metadata">

**Author:** ![setiseta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/setiseta/32/29949_2.png) [@setiseta](https://discuss.elastic.co/u/setiseta)\
**Post date:** [June 12, 2018, 11:13am UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/5 "2018-06-12T11:13:57Z")

</div>

hello,  
i have a similar problem but with filebeat.

```auto
2018-06-12T11:04:48.747Z ERROR pipeline/output.go:92 Failed to publish events: temporary bulk send failure
2018-06-12T11:04:48.749Z INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.4
2018-06-12T11:04:48.751Z INFO template/load.go:73 Template already exists and will not be overwritten.

```

the config:

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            or:
              - contains:
                  docker.container.image: tomcat
              - equals:
                  docker.container.labels.com.docker.compose.service: tomcat
              - contains:
                  docker.container.name: tomcat
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}|(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \d\d)'
              multiline.negate: true
              multiline.match: after
              fields:
                autodiscover: 'image-tomcat'
              pipeline: tomcat_level
        - condition:
            and:
              - not:
                contains:
                  docker.container.image: tomcat
              - not:
                equals:
                  docker.container.labels.com.docker.compose.service: tomcat
              - not:
                contains:
                  docker.container.name: tomcat
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              fields:
                autodiscover: 'default'

fields:
  env: 'dev'

setup.kibana:
  host: "https://kibana.local:443"

output.elasticsearch:
  hosts: ["localhost:9200"]
  worker: 8
  index: dev-%{[beat.version]}-%{+yyyy.MM.dd}

setup.template.name: "dev"
setup.template.pattern: "dev-*"
setup.dashboards.index: "dev-*"

```

any hints whats wrong here?

---

<div class="post-metadata">

**Author:** ![nuwancs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nuwancs/32/46419_2.png) [@nuwancs](https://discuss.elastic.co/u/nuwancs)\
**Post date:** [June 12, 2018, 6:10pm UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/6 "2018-06-12T18:10:55Z")

</div>

Hi @setiseta  
I am pretty sure this is due using a custom index name. try removing custom index name  
Remove  
`index: dev-%{[beat.version]}-%{+yyyy.MM.dd}`

---

<div class="post-metadata">

**Author:** ![setiseta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/setiseta/32/29949_2.png) [@setiseta](https://discuss.elastic.co/u/setiseta)\
**Post date:** [June 13, 2018, 9:50am UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/7 "2018-06-13T09:50:14Z")

</div>

No change with the index name to default.  
But i think i've found it.  
I've change now from:  
`DOCKER_OPTS="--log-opt max-size=100m --log-opt max-file=5"`  
to:  
`DOCKER_OPTS="--log-opt max-size=500m"`

for now it is gone.  
it's a setting to protect the server of disk space problems due to docker logging.  
(default is unlimited logging size, and with the file num it rolls files)  
maybe there is a bug / unexpected behavior of filebeat if logging options are set to create some files.

> **[JSON File logging driver](https://docs.docker.com/config/containers/logging/json-file/#options)**
>
> By default, Docker captures the standard output (and standard error) of all your containers, and writes them in files using the JSON format. The JSON format annotates each line with...

maybe only with autodiscovery which is experimental.

---

<div class="post-metadata">

**Author:** ![setiseta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/setiseta/32/29949_2.png) [@setiseta](https://discuss.elastic.co/u/setiseta)\
**Post date:** [June 19, 2018, 6:00am UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/8 "2018-06-19T06:00:33Z")

</div>

the errors occoures again after some time.  
any more hints?  
is it the log file size maybe?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2018, 6:00am UTC](https://discuss.elastic.co/t/failed-to-publish-events-temporary-bulk-send-failure/134330/9 "2018-07-17T06:00:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.