# ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
**URL:** https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361
**Category:** Elasticsearch
**Created:** [February 21, 2019, 9:46am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361 "2019-02-21T09:46:23Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 21, 2019, 9:46am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/1 "2019-02-21T09:46:23Z")
I am getting the error whenever I try ingesting data into elasticsearch using filebeat.
```
2019-02-21T07:57:02.190Z INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.5.4
2019-02-21T07:57:02.193Z INFO template/load.go:130 Template already exists and will not be overwritten.
2019-02-21T07:57:02.193Z INFO instance/beat.go:894 Template successfully loaded.
2019-02-21T07:57:02.193Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://localhost:9200)) established
2019-02-21T07:57:04.513Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
2019-02-21T07:57:04.513Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://localhost:9200))
2019-02-21T07:57:04.515Z INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.5.4
2019-02-21T07:57:04.519Z INFO template/load.go:130 Template already exists and will not be overwritten.
```
This is the `pipeline.json` used for defining pipeline:
```
{
"description": "Pipeline for ingest node",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{IP:source_ip} %{GREEDYDATA} \\[%{HTTPDATE:request_date}\\] \\\"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\\\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} \\\"%{GREEDYDATA:user_agent}\\\" %{GREEDYDATA:x_play_back_session_id}",
"%{IP:source_ip} %{GREEDYDATA} \\[%{HTTPDATE:request_date}\\] \\\"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\\\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent:int} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} %{GREEDYDATA:user_agent}"
],
"on_failure": [
{
"grok": {
"field": "message",
"patterns": ["%{IP:source_ip} %{GREEDYDATA} \\[%{HTTPDATE:request_date}\\] \\\"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\\\" %{NUMBER:response_code} %{NUMBER:bytes_sent} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} %{GREEDYDATA:user_agent}"]
}
}
]
}
},
{
"convert": {
"field": "bytes_sent",
"type": "integer"
}
},
{
"dissect": {
"field": "uri_path",
"if": "(ctx.uri_path.contains(\"hls5\") && ctx.uri_path.contains(\"live\") && (ctx.uri_path.contains(\"m3u8\") || ctx.uri_path.contains(\"ts\"))) || (ctx.uri_path.contains(\"dash\") && ctx.uri_path.contains(\"live\") && ctx.uri_path.contains(\"m4s\"))",
"pattern": "/%{a}/%{protocol}/%{stream_type}/%{backend_channel_id}/%{e}/%{variant}/%{g}.%{h}"
}
},
{
"remove": {
"field": [
"a",
"e",
"g",
"h"
]
}
}
]
}
```
This is the curl command using `pipeline.json` for PUT PIPELINE API:
`curl -H 'Content-Type: application/json' -X PUT 'localhost:9200/_ingest/pipeline/test-pipeline' -d@pipeline.json`
This is the `filebeat.yml` file.
```
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/dump/log/*
exclude_lines:
- thumbnail
- pictures
- health
- stats
- alerts
- url_template
- resource
- config
include_lines:
- live
- vod
- data
output.elasticsearch:
hosts: ["localhost:9200"]
index: "test-pipeline-%{+yyyy.MM.dd}"
pipeline: "test-pipeline"
setup.template.name: "test-pipeline"
setup.template.pattern: "test-pipeline*"
setup:
kibana:
host: "localhost:5601"
dashboards:
index: "test-pipeline*"
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 21, 2019, 9:50am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/2 "2019-02-21T09:50:42Z")
You have named your pipeline `ats-pipeline` but configured `test-pipeline` in the Filebeat config.
---
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 21, 2019, 4:13pm UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/4 "2019-02-21T16:13:11Z")
My mistake. Was a typo. Edited the topic to the correct value. The error still gets generated about the bulk send failure.
---
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 22, 2019, 5:16am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/5 "2019-02-22T05:16:05Z")
Any suggestions??
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2019, 7:48am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/6 "2019-02-22T07:48:05Z")
I do not see anything else obviously wrong. Can you increase the Filebeat logging level? Is there anything in the Elasticsearch logs? Have you tested your pipeline for different types of records using the simulate API?
---
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 22, 2019, 9:26am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/7 "2019-02-22T09:26:28Z")
Using the simulate API, i have tested both correct and erroneous logs which are supposed to be ingested. They're giving the correct results.
Elasticsearch only shows this :
```
[2019-02-22T09:21:49,518][WARN][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] [inkba01p1] Deprecated field [template] used, replaced by [index_patterns]
[2019-02-22T09:21:50,533][WARN][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] [inkba01p1] Deprecated field [template] used, replaced by [index_patterns]
[2019-02-22T09:21:51,545][WARN][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] [inkba01p1] Deprecated field [template] used, replaced by [index_patterns]
```
i haven't set logging.level yet. will check.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2019, 9:29am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/8 "2019-02-22T09:29:55Z")
It seems that the template you are trying to apply is not in the correct format. Was this by any chance created for an older version of Elasticsearch?
---
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 22, 2019, 9:33am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/9 "2019-02-22T09:33:51Z")
This is how I have set the templates:
```
setup.template.name: "test-pipeline"
setup.template.pattern: "test-pipeline*"
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2019, 9:38am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/10 "2019-02-22T09:38:36Z")
I am talking about the format of the template itself, not how you specify it in the Filebeat config.
---
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 22, 2019, 10:29am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/11 "2019-02-22T10:29:43Z")
Yeah. Saw it. It was created for an older version of Elasticsearch. Changed it to the latest one. Still nothing.
Elasticsearch logs show:
`[2019-02-22T15:58:31,384][INFO][o.e.c.m.MetaDataIndexTemplateService] [SHARADA-LAPTOP] adding template [kibana_index_template:.kibana] for index patterns [.kibana]`
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2019, 10:48am UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/12 "2019-02-22T10:48:24Z")
Do you have any non-default settings in your Elasticsearch configuration?
---
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 22, 2019, 12:12pm UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/13 "2019-02-22T12:12:19Z")
Nope. All are the default ones.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2019, 12:18pm UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/14 "2019-02-22T12:18:00Z")
Did you get any additional information from increasing the log level to DEBUG?
---
### Author:  [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)
#### Post date: [February 22, 2019, 12:23pm UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/15 "2019-02-22T12:23:48Z")
Nope. I am receiving this.
```
2019-02-22T17:51:18.121+0530 DEBUG [input] log/input.go:174 Start next scan
2019-02-22T17:51:18.124+0530 DEBUG [input] log/input.go:404 Check file for harvesting: /var/dump/log/ats1.log
2019-02-22T17:51:18.125+0530 DEBUG [input] log/input.go:494 Update existing file for harvesting: /var/dump/log/ats1.log, offset: 439830
2019-02-22T17:51:18.125+0530 DEBUG [input] log/input.go:546 Harvester for file is still running: /var/dump/log/ats1.log
2019-02-22T17:51:18.125+0530 DEBUG [input] log/input.go:195 input states cleaned up. Before: 1, After: 1, Pending: 0
2019-02-22T17:51:23.087+0530 DEBUG [harvester] log/log.go:102 End of file reached: /var/dump/log/ats1.log; Backoff now.
2019-02-22T17:51:28.127+0530 DEBUG [input] input/input.go:152 Run input
2019-02-22T17:51:28.127+0530 DEBUG [input] log/input.go:174 Start next scan
2019-02-22T17:51:28.131+0530 DEBUG [input] log/input.go:404 Check file for harvesting: /var/dump/log/ats1.log
2019-02-22T17:51:28.131+0530 DEBUG [input] log/input.go:494 Update existing file for harvesting: /var/dump/log/ats1.log, offset: 439830
2019-02-22T17:51:28.132+0530 DEBUG [input] log/input.go:546 Harvester for file is still running: /var/dump/log/ats1.log
2019-02-22T17:51:28.133+0530 DEBUG [input] log/input.go:195 input states cleaned up. Before: 1, After: 1, Pending: 0
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 22, 2019, 12:23pm UTC](https://discuss.elastic.co/t/error-pipeline-output-go-121-failed-to-publish-events-temporary-bulk-send-failure/169361/16 "2019-03-22T12:23:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.