# Filebeat failed to publish events: temporary bulk send failure

**URL:** <https://discuss.elastic.co/t/filebeat-failed-to-publish-events-temporary-bulk-send-failure/288566>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [November 7, 2021, 4:06pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-publish-events-temporary-bulk-send-failure/288566 "2021-11-07T16:06:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cdalexndr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdalexndr/32/55274_2.png) [@cdalexndr](https://discuss.elastic.co/u/cdalexndr)\
**Post date:** [November 7, 2021, 4:06pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-publish-events-temporary-bulk-send-failure/288566/1 "2021-11-07T16:06:49Z")

</div>

Filebeat 7.14.1 is stuck in a log loop with `failed to publish events: temporary bulk send failure`.

Turning debug log, it shows:

```auto
filebeat | 2021-11-07T15:57:49.938Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
filebeat | 2021-11-07T15:57:49.941Z DEBUG [elasticsearch] elasticsearch/client.go:227 PublishEvents: 1 events have been published to elasticsearch in 3.1342ms.
filebeat | 2021-11-07T15:57:49.941Z DEBUG [elasticsearch] elasticsearch/client.go:411 Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
filebeat | 2021-11-07T15:57:49.941Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
filebeat | 2021-11-07T15:57:49.941Z INFO [publisher] pipeline/retry.go:223 done
filebeat | 2021-11-07T15:57:51.776Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure

```

This error is so criptic with missing information that I don't know how to get more information...  
What String caused error, doing what???  
I've searched Elasticsearch logs for `string_index_out_of_bounds_exception` but didn't found anything.

My filebeat.yml:

```auto
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "log-%{[docker][container][labels][app]}"
  pipeline: pipeline_logs
  ilm.enabled: false

logging.level: debug

queue.mem:
  events: 1024
  flush.min_events: 256
  flush.timeout: 1s

setup.ilm.enabled: false
setup.template.enabled: false
setup.kibana.host: http://kibana:5601

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      templates:
        - condition:
            contains:
              docker.container.image: swag
          config:
            - module: nginx
              access:
                enabled: true
                var.paths: ["/config/log/nginx/access.log*"]
              error:
                enabled: true
                var.paths: ["/config/log/nginx/error.log*"]

```

My pipeline\_logs has an `on_failure` fallback processor, but it seems the problem is somewhere else:

```auto

{
  "description": "Log dispatcher ingest pipeline",
  "processors": [
    {
      "set": {
        "field": "app_name",
        "value": "{{docker.container.labels.app}}",
        "ignore_empty_value" : true
      }
    },
    {
      "pipeline": {
        "if": "ctx.app_name == 'autoheal'",
        "name": "pipeline_autoheal"
      }
    },
    {
      "pipeline": {
        "if": "ctx.app_name == 'web'",
        "name": "pipeline_web"
      }
    },
    {
      "pipeline": {
        "if": "ctx.app_name == null",
        "name": "pipeline_fallback"
      }
    },
    {
      "uppercase": {
        "field": "log.level",
        "if": "ctx.log?.level != null"
      }
    },
    {
      "set": {
        "field": "level",
        "value": "{{log.level}}",
        "if": "ctx.log?.level != null"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "description": "Record error information",
        "field": "error",
        "value": "Processor '{{ _ingest.on_failure_processor_type }}' with tag '{{ _ingest.on_failure_processor_tag }}' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message: {{ _ingest.on_failure_message }}"
      }
    }
  ]
}

```

I've done some pipeline tests in Kibana and it seems to be working fine.

---

<div class="post-metadata">

**Author:** ![cdalexndr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdalexndr/32/55274_2.png) [@cdalexndr](https://discuss.elastic.co/u/cdalexndr)\
**Post date:** [November 7, 2021, 4:34pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-publish-events-temporary-bulk-send-failure/288566/2 "2021-11-07T16:34:56Z")

</div>

I've narrowed it down to a container (mariadb:latest) that when running, causes filebeat to stop working.

But I've run all my mariadb container log lines through my pipeline (Kibana test pipeline) and none failed...

My mariadb container has no labels for filebeat configuration.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 5, 2021, 6:35pm UTC](https://discuss.elastic.co/t/filebeat-failed-to-publish-events-temporary-bulk-send-failure/288566/3 "2021-12-05T18:35:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.