Send message failed from dedicated ingest nodes to masters 7.3.1

We have 3 dedicated ingest nodes in a cluster. They are functioning and data is coming into ES, but they are logging constantly "send messsage failed". No idea why ... logs about every few seconds from each ingest node and each master. Data nodes don't log it and coordinator nodes don't log it. Just the ingest nodes.

Full log output is:

2019-09-06T16:52:57,089][WARN ][o.e.t.OutboundHandler    ] [ingest.clustername] send message failed [channel: Netty4TcpChannel{localAddress=/10.10.10.5:9300, remoteAddress=/10.10.10.1:44628}]
java.lang.NullPointerException: null
	at org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:403) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.DocWriteResponse.writeTo(DocWriteResponse.java:283) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.update.UpdateResponse.writeTo(UpdateResponse.java:82) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.bulk.BulkItemResponse.writeTo(BulkItemResponse.java:510) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.action.bulk.BulkResponse.writeTo(BulkResponse.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:70) ~[elasticsearch-7.3.1.jar:7.3.1]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:53) ~[elasticsearch-7.3.1.jar:7.3.1]

These logs were on both the masters and the dedicated ingest nodes. No issues on dedicated data nodes or coordinator nodes.
Stopping FileBeat on the master nodes stopped the errors.
FileBeat has no issues anywhere else and the data was ingested so no idea what the problem is.

Filebeat was using the localhost as it's ES setting. Going to try some things and see if I can figure out why Filebeat on the masters is an issue.

Changing the output for Filebeat on the masters worked:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["ingest-node-fqdn:9200"]

So data nodes, coordinator nodes and ingest nodes had no problem with localhost:9200 but master only nodes caused errors between each master and the ingest node when using localhost.

Smells like a bug ...

Yep, it's a bug https://github.com/elastic/elasticsearch/issues/46678

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.