# Error: failed to publish events: temporary bulk send failure
**URL:** https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270
**Category:** Beats
**Tags:** filebeat
**Created:** [March 9, 2022, 11:33pm UTC](https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270 "2022-03-09T23:33:51Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@samettozbay](https://discuss.elastic.co/u/samettozbay)
#### Post date: [March 9, 2022, 11:33pm UTC](https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270/1 "2022-03-09T23:33:51Z")
I have issue about sending logs with filebeat. I have checked all the topics related this issue but i couldn't figure out it yet .
I have Elasticsearch v 7.10.2 and i am using filebeat 7.10.0.
I created my pipeline as below. when i trying to simulate pipeline with my logs, it is succesfull. ı thınk ı have no problem with my grok pattern .
```auto
{ -
"description": " XXXX",
"processors": [ -
{ -
"grok": { -
"field": "message",
"patterns": [ -
"%{SYSLOGBASE2} trx_id:%{GREEDYDATA:trx_id} %{GREEDYDATA:program} trxid:%{GREEDYDATA:trxid}, appId:%{GREEDYDATA:appId}, serviceMethod:%{GREEDYDATA:serviceMethod}, serviceId:%{GREEDYDATA:serviceId}, accessedCountryCode2:%{WORD:accessedCountryCode2}, username:%{USERNAME:username}, schemaName:%{WORD:schemaName}, regionCode2:%{WORD:regionCode2}, clientip:%{IPORHOST:clientip}|%{GREEDYDATA:clientip}, failureText: %{WORD:failureText}, sessionId:%{GREEDYDATA:sessionId}, detailMessage:%{GREEDYDATA:detailMessage}"
]
}
},
{ -
"date": { -
"field": "timestamp8601",
"formats": [ -
"yyyy-MM-dd HH:mm:ss.SSS"
]
}
},
{ -
"date_index_name": { -
"field": "timestamp8601",
"index_name_prefix": "XXXXX-",
"index_name_format": "yyyy-MM-dd-HH",
"timezone": "UTC+3",
"date_rounding": "d",
"date_formats": [ -
"yyyy-MM-dd HH:mm:ss.SSS"
]
}
}
]
}
}
```
my problem is starting with filebeat. When i started filebeat to send logs es, i could't send to all logs to es. (maybe %5 logs successfully sending)
filebeat yml file:
```auto
filebeat.inputs:
- type: log
enabled: true
paths:
- /......./logs/app.log
multiline.pattern: \d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d{3}
multiline.negate: true
multiline.match: after
filebeat.config.modules:
path:
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.ilm.enabled: false
output.elasticsearch:
hosts: ["xx.xx.x.xxx", "xx.xx.x.xxx", "xx.xx.x.xxx"]
protocol: "http"
username: "xxx"
password: "xxxx"
ssl.verification_mode: none
loadbalance: true
pipeline: "xxxx"
bulk_max_size: 256
worker: 4
compression_level: 0
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
```
here is the filebeatlogs:
2022-03-10T00:16:20.263+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:20.263+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:20.421+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:20.421+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:21.290+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:21.290+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:21.986+0300 ERROR [publisher\_pipeline\_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
2022-03-10T00:16:22.290+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.290+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.376+0300 ERROR [publisher\_pipeline\_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.401+0300 INFO [publisher\_pipeline\_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch([http://xx.xx.x.xxx:9200](http://xx.xx.x.xxx:9200)))
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.401+0300 INFO [publisher\_pipeline\_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch([http://xx.xx.x.xxx:9200](http://xx.xx.x.xxx:9200)))
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.402+0300 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2022-03-10T00:16:22.403+0300 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2022-03-10T00:16:22.406+0300 INFO template/load.go:97 Template filebeat-7.10.0 already exists and will not be overwritten.
2022-03-10T00:16:22.406+0300 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2022-03-10T00:16:22.407+0300 INFO [publisher\_pipeline\_output] pipeline/output.go:151 Connection to backoff(Elasticsearch([http://xx.xx.x.xxx:9200](http://xx.xx.x.xxx:9200))) established
2022-03-10T00:16:22.453+0300 INFO template/load.go:97 Template filebeat-7.10.0 already exists and will not be overwritten.
2022-03-10T00:16:22.453+0300 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2022-03-10T00:16:22.454+0300 INFO [publisher\_pipeline\_output] pipeline/output.go:151 Connection to backoff(Elasticsearch([http://xx.xx.x.xxx:9200](http://xx.xx.x.xxx:9200))) established
2022-03-10T00:16:22.643+0300 ERROR [publisher\_pipeline\_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
anyone any idea ?
thanks.
---
### Author:  [@samettozbay](https://discuss.elastic.co/u/samettozbay)
#### Post date: [March 11, 2022, 7:39am UTC](https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270/2 "2022-03-11T07:39:37Z")
any idea ?
---
### Author:  [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)
#### Post date: [March 14, 2022, 1:54pm UTC](https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270/3 "2022-03-14T13:54:20Z")
Hi @samettozbay
Could you please explain why you defined `bulk_max_size` and `worker`, was there any specific need for this?:
```auto
hosts: ["xx.xx.x.xxx", "xx.xx.x.xxx", "xx.xx.x.xxx"]
loadbalance: true
bulk_max_size: 256
worker: 4
```
Did you try to use default configuration? Do you see the same behavior?
According to: [Configure the Elasticsearch output | Filebeat Reference [7.10] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.10/elasticsearch-output.html#_worker) - you are starting 12 workers, 4 for each instance - do you maybe see an increase of resource usage of Elasticsearch instances, that could cause connectivity issues?
regarding [`bulk_max_size`](https://www.elastic.co/guide/en/beats/filebeat/7.10/elasticsearch-output.html#_bulk_max_size):
```auto
However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.
```
---
### Author:  [@samettozbay](https://discuss.elastic.co/u/samettozbay)
#### Post date: [March 15, 2022, 11:17am UTC](https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270/4 "2022-03-15T11:17:18Z")
Hi @Tetiana_Kravchenko
Thanks for reply. I have tried default configuration but nothing changed. Same result.
---
### Author:  [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)
#### Post date: [March 16, 2022, 9:00am UTC](https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270/5 "2022-03-16T09:00:39Z")
@samettozbay Did you check resource usage of Elasticsearch instances? could it be an issue?
Any relevant Elasticsearch logs?
Also did you try to connect to elasticsearch\_ip:port? maybe some network limitations on the Elasticsearch instances?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 13, 2022, 9:00am UTC](https://discuss.elastic.co/t/error-failed-to-publish-events-temporary-bulk-send-failure/299270/6 "2022-04-13T09:00:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.