First we were using filebeat version 6.2.4 in which filebeat automatically exported 'host' field as string but when we updated filebeat to 6.4.2, newer version started exporting 'host' field as an object and this led us data loss as elasticsearch would not allow an object in string typed field. We create indexes on daily basis so basically, if We upgrade filebeat today then We would lose all data until new mapping gets created on tomorrow's index.
How can we avoid this? We tried dropping fields using drop processor in filebeat but seems It would only drop specific field not the whole object.
Are you using filebeat with logstash? The host field used to be set in logstash.
In 6.4, beats introduced the host namespace. But logstash used to add a host field to events received.
How can we avoid this?
In beats we normally add the beat version to index names. That is, the default filebeat index is filebeat-<version>-<date>. By versioning indices and template mappings we can at least continue to index new data if the mappings would be incompatible.
If you use logstash or ingest node, then you have update the mappings in there.
Are you using filebeat with logstash? The host field used to be set in logstash.
In 6.4, beats introduced the host namespace. But logstash used to add a host field to events received.
Yes, we are using filebeat with logstash.
We have already created separate indices per teams and due to the older version host field is already set up by logstash which is string. Now if we upgrade filebeat then it will start sending out host object which would create rejection on elastic side.
Can we disable that host object on filebeat side? so even after the upgrade, everything works normally as before. or we just have to put a filter to drop that field on logstash side?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.