Filebeat incompatibility when upgrading elasticsearch

[bruce289]

Hi.

We're currently testing an upgrade from Elasticsearch 6.8 to 7.15 and we noticed that when we upgrade the cluster, the Filebeat agents on some of our hosts can no longer write to the indices and we see the errors like the following in the Filebeat logs:

Rejecting mapping update to [testindex-2020.11.30] as the final mapping would have more than 1 type: [doc, _doc]

It seems that if Filebeat has been restarted after the cluster has been upgraded or if this is a newly built host, Filebeat detects that the cluster is now v7 and starts indexing events with a document type of _doc, whereas on hosts where Filebeat hasn't restarted, it still thinks that the cluster is v6 and still indexes events with a document type of doc.

I must be overlooking something here but I can't figure out how we're going to do this other than to restart filebeat at midnight on all our hosts after the upgrade is completed, but even that isn't that easy at our scale. Would really appreciate some ideas here. Thanks.

[warkolm (Mark Walkom)]

Yeah that's a bit of a hassle, as we changed to doc, only to settle on _doc

It might be worth upgrading all your Filebeat instances first, then Elasticsearch next. What sort of scale are you working at?

[bruce289]

Thanks for the reply, Mark. Though I can't see how that would help us. Filebeat 7 will still automatically detect the version of the cluster and adjust the document type accordingly.