# Send message failed from dedicated ingest nodes to masters 7.3.1

**URL:** https://discuss.elastic.co/t/send-message-failed-from-dedicated-ingest-nodes-to-masters-7-3-1/198454
**Category:** Elasticsearch
**Created:** [September 6, 2019, 5:11pm UTC](https://discuss.elastic.co/t/send-message-failed-from-dedicated-ingest-nodes-to-masters-7-3-1/198454 "2019-09-06T17:11:28Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [September 6, 2019, 5:11pm UTC](https://discuss.elastic.co/t/send-message-failed-from-dedicated-ingest-nodes-to-masters-7-3-1/198454/1 "2019-09-06T17:11:28Z")

</div>

We have 3 dedicated ingest nodes in a cluster. They are functioning and data is coming into ES, but they are logging constantly "send messsage failed". No idea why ... logs about every few seconds from each ingest node and each master. Data nodes don't log it and coordinator nodes don't log it. Just the ingest nodes.

Full log output is:

```
2019-09-06T16:52:57,089][WARN][o.e.t.OutboundHandler] [ingest.clustername] send message failed [channel: Netty4TcpChannel{localAddress=/10.10.10.5:9300, remoteAddress=/10.10.10.1:44628}]
java.lang.NullPointerException: null
	at org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:403) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.DocWriteResponse.writeTo(DocWriteResponse.java:283) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.update.UpdateResponse.writeTo(UpdateResponse.java:82) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.bulk.BulkItemResponse.writeTo(BulkItemResponse.java:510) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.bulk.BulkResponse.writeTo(BulkResponse.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:70) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:53) ~[elasticsearch-7.3.1.jar:7.3.1]
```

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [September 11, 2019, 12:43pm UTC](https://discuss.elastic.co/t/send-message-failed-from-dedicated-ingest-nodes-to-masters-7-3-1/198454/2 "2019-09-11T12:43:57Z")

</div>

These logs were on both the masters and the dedicated ingest nodes. No issues on dedicated data nodes or coordinator nodes.  
Stopping FileBeat on the master nodes stopped the errors.  
FileBeat has no issues anywhere else and the data was ingested so no idea what the problem is.

Filebeat was using the localhost as it's ES setting. Going to try some things and see if I can figure out why Filebeat on the masters is an issue.

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [September 11, 2019, 12:57pm UTC](https://discuss.elastic.co/t/send-message-failed-from-dedicated-ingest-nodes-to-masters-7-3-1/198454/3 "2019-09-11T12:57:51Z")

</div>

Changing the output for Filebeat on the masters worked:

```auto
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["ingest-node-fqdn:9200"]

```

So data nodes, coordinator nodes and ingest nodes had no problem with `localhost:9200` but master only nodes caused errors between each master and the ingest node when using localhost.

Smells like a bug ...

---

<div class="post-metadata">

### Author: ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [September 12, 2019, 6:03pm UTC](https://discuss.elastic.co/t/send-message-failed-from-dedicated-ingest-nodes-to-masters-7-3-1/198454/4 "2019-09-12T18:03:18Z")

</div>

Yep, it's a bug [https://github.com/elastic/elasticsearch/issues/46678](https://github.com/elastic/elasticsearch/issues/46678)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 10, 2019, 6:09pm UTC](https://discuss.elastic.co/t/send-message-failed-from-dedicated-ingest-nodes-to-masters-7-3-1/198454/5 "2019-10-10T18:09:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
