# Filebeat modules config: failed to publish events: temporary bulk send failure

**URL:** <https://discuss.elastic.co/t/filebeat-modules-config-failed-to-publish-events-temporary-bulk-send-failure/284948>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [September 22, 2021, 10:20pm UTC](https://discuss.elastic.co/t/filebeat-modules-config-failed-to-publish-events-temporary-bulk-send-failure/284948 "2021-09-22T22:20:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cdalexndr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdalexndr/32/55274_2.png) [@cdalexndr](https://discuss.elastic.co/u/cdalexndr)\
**Post date:** [September 22, 2021, 10:20pm UTC](https://discuss.elastic.co/t/filebeat-modules-config-failed-to-publish-events-temporary-bulk-send-failure/284948/1 "2021-09-22T22:20:37Z")

</div>

Just upgraded elastic stack from 7.10.1 to 7.14.1, and now filebeat doesn't work!  
Filebeat logs repeats the following lines:

```auto
filebeat | 2021-09-22T21:57:46.145Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
filebeat | 2021-09-22T21:57:46.278Z INFO template/load.go:111 Template "log" already exists and will not be overwritten.
filebeat | 2021-09-22T21:57:46.278Z INFO [index-management] idxmgmt/std.go:297 Loaded index template.
filebeat | 2021-09-22T21:57:46.280Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
filebeat | 2021-09-22T21:57:46.332Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
filebeat | 2021-09-22T21:57:46.332Z INFO [publisher] pipeline/retry.go:223 done
filebeat | 2021-09-22T21:57:47.527Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
filebeat | 2021-09-22T21:57:47.528Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://elasticsearch:9200))
filebeat | 2021-09-22T21:57:47.528Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
filebeat | 2021-09-22T21:57:47.528Z INFO [publisher] pipeline/retry.go:223 done

```

filebeat.yml:

```auto
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "log-%{[docker][container][labels][app]}"
  pipeline: pipeline_logs
  ilm.enabled: false

queue.mem:
  events: 1024
  flush.min_events: 256
  flush.timeout: 1s

setup.ilm.enabled: false
setup.template.name: "log"
setup.template.pattern: "log-*"
setup.kibana.host: http://kibana:5601

filebeat.modules:
  - module: system
    syslog:
      enabled: true
      var.paths: ["/hostfs/var/log/syslog*"]

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      templates:
        - condition:
            contains:
              docker.container.image: swag
          config:
            - module: nginx
              access:
                enabled: true
                var.paths: ["/config/log/nginx/access.log*"]
              error:
                enabled: true
                var.paths: ["/config/log/nginx/error.log*"]

```

filebeat docker-compose service:

```auto
  filebeat:
    image: store/elastic/filebeat:7.14.1
    container_name: filebeat
    mem_limit: 128m
    entrypoint: /usr/share/filebeat/config/init/entry.sh
    volumes:
      - ./config/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./config/filebeat/init:/usr/share/filebeat/config/init:ro
      - ./config/filebeat/pipelines:/usr/share/filebeat/config/pipelines:ro
      - ./config/filebeat/index:/usr/share/filebeat/config/index:ro
      - /var/lib/docker/containers/:/var/lib/docker/containers/:ro #read logs from containers
      - /var/run/docker.sock:/var/run/docker.sock:ro #docker daemon listener
      - filebeat-data:/usr/share/filebeat/data
      - entry-data:/config:ro
      - /var/log:/hostfs/var/log:ro
    user: root
    labels:
      app: filebeat
    logging:
      driver: 'json-file'
      options:
        max-size: '20m'
        max-file: '5'
        compress: 'true'

```

Elasticsearch logs shows a warning related to filebeat

> docker logs elasticsearch --tail 1000 | grep filebeat

```auto
{
    "type": "server",
    "timestamp": "2021-09-22T21:08:34,295Z",
    "level": "WARN",
    "component": "o.e.c.m.MetadataIndexTemplateService",
    "cluster.name": "docker-cluster",
    "node.name": "036312a73667",
    "message": "index template [log-filebeat-template] has index patterns [log-filebeat] matching patterns from existing older templates [log] with patterns (log => [log-*]); this template [log-filebeat-template] will take precedence during new index creation",
    "cluster.uuid": "Uvr-gj7UQx-CGv-lBaMXUw",
    "node.id": "6ego0coRQdS96SIFGP7mMQ"
}

```

There are no ERRORS in elasticsearch logs.

Note that metricbeat works as expected, metrics are stored and can be retrieved.

---

<div class="post-metadata">

**Author:** ![cdalexndr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdalexndr/32/55274_2.png) [@cdalexndr](https://discuss.elastic.co/u/cdalexndr)\
**Post date:** [September 22, 2021, 11:05pm UTC](https://discuss.elastic.co/t/filebeat-modules-config-failed-to-publish-events-temporary-bulk-send-failure/284948/2 "2021-09-22T23:05:39Z")

</div>

It seems that the `log` index template was made by the old version and filebeat will not change it.  
I've deleted it and changed `filebeat.yml` to include filebeat version:

```auto
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "log-%{[agent.version]}-%{[docker][container][labels][app]}"
 ....
setup.template.name: "log-%{[agent.version]}"
setup.template.pattern: "log-%{[agent.version]}-*"

```

Now the same issue, just logging changed to `Template "log-7.14.1"`:

```auto
filebeat | 2021-09-22T23:01:48.746Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
filebeat | 2021-09-22T23:01:48.784Z INFO template/load.go:111 Template "log-7.14.1" already exists and will not be overwritten.
filebeat | 2021-09-22T23:01:48.784Z INFO [index-management] idxmgmt/std.go:297 Loaded index template.
filebeat | 2021-09-22T23:01:48.814Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
filebeat | 2021-09-22T23:01:48.830Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
filebeat | 2021-09-22T23:01:48.830Z INFO [publisher] pipeline/retry.go:223 done
filebeat | 2021-09-22T23:01:50.039Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
filebeat | 2021-09-22T23:01:50.039Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://elasticsearch:9200))
filebeat | 2021-09-22T23:01:50.039Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
filebeat | 2021-09-22T23:01:50.040Z INFO [publisher] pipeline/retry.go:223 done

```

Removed template index altogether with `setup.template.enabed: false` and same problem:

```auto
filebeat | 2021-09-22T23:15:46.441Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
filebeat | 2021-09-22T23:15:46.540Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
filebeat | 2021-09-22T23:15:46.540Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
filebeat | 2021-09-22T23:15:46.540Z INFO [publisher] pipeline/retry.go:217 done
filebeat | 2021-09-22T23:15:47.748Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
filebeat | 2021-09-22T23:15:47.749Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://elasticsearch:9200))
filebeat | 2021-09-22T23:15:47.749Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
filebeat | 2021-09-22T23:15:47.749Z INFO [publisher] pipeline/retry.go:217 done

```

---

<div class="post-metadata">

**Author:** ![cdalexndr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdalexndr/32/55274_2.png) [@cdalexndr](https://discuss.elastic.co/u/cdalexndr)\
**Post date:** [September 23, 2021, 1:01pm UTC](https://discuss.elastic.co/t/filebeat-modules-config-failed-to-publish-events-temporary-bulk-send-failure/284948/3 "2021-09-23T13:01:06Z")

</div>

Elasticsearch is accessible from filebeat container:

```auto
curl -XGET http://elasticsearch:9200/_cluster/health
{"cluster_name":"docker-cluster","status":"green","...

```

Following the troubleshooting guide, I run filebeat with `filebeat -e -d "publisher"`, but no new logs are written.  
Runing filebeat as `filebeat -e -d "*"` becomes unusable, too many logs in console.

Tried to replicate this upgrade issue in a dev environment, but there it worked fine.  
This issue happened before, in a previous update, but I didn't had much data so I just deleted all old data for filebeat to work.

Any pointers how to investigate this problem? I don't want to loose data after every update!

---

<div class="post-metadata">

**Author:** ![cdalexndr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cdalexndr/32/55274_2.png) [@cdalexndr](https://discuss.elastic.co/u/cdalexndr)\
**Post date:** [September 23, 2021, 4:06pm UTC](https://discuss.elastic.co/t/filebeat-modules-config-failed-to-publish-events-temporary-bulk-send-failure/284948/4 "2021-09-23T16:06:07Z")

</div>

With the version update, I've also added a `filebeat.modules:` section to configuration, that seems to cause this problem.  
If I remove the `filebeat.modules:` section from my configuration, then filebeat works.  
Opened github issue [https://github.com/elastic/beats/issues/28100](https://github.com/elastic/beats/issues/28100)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 21, 2021, 6:06pm UTC](https://discuss.elastic.co/t/filebeat-modules-config-failed-to-publish-events-temporary-bulk-send-failure/284948/5 "2021-10-21T18:06:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.