@ruflin our use case is closest to "You use a custom template and your indices are not versioned".
The main issue here is that we already use a
host field, and have done for several years. We are shipping data from several hundred hosts, spanning many different clients, and with the mix of hosts and different naming conventions that our clients have, we invariably have a number of short hostname collisions across different systems, e.g., prod-web1, etc.
These hostname collisions necessitated the use of a custom field containing FQDNs for all hosts; this being the
host field, and we've been doing it this way since well before Beats existed (i.e., Logstash Forwarder, Lumberjack and Log Courier).
My initial tests with Filebeat 6.3.0 seem to indicate that we can't even ship a custom host field any more - it's simply clobbered by the host object in this version, thus we can't identify systems by their FQDN host field when using Beats 6.3.0. This is a major problem for us.
Adding to the problem is we have clients who are required to retain indices for long periods for PCI compliance reasons. If we attempted to change this mapping, we couldn't simply wait for their affected indices to roll over because of the retention periods these customers are required to have. Changing critical mappings such as the host field will also have an ongoing effect for any of their historical data, as there will be mapping conflicts lasting for months or years. Not being able to aggregate data based on host will be entirely unacceptable to these clients.
Our Logstash pipeline refers to the host field in a number of places, as do the majority of our Kibana dashboards. Many of our custom systems which query our log data from outside the Elastic Stack (monitoring, metrics, alerting, etc) also refer to the host field. Changing this mapping means we would need to modify almost everything that touches our Elastic Stack. Coordinating such a change would be a nightmare.
I am sure we are not alone in using a field named
host to store hostname data. This seemingly simple change breaks everything we have been building over the past few years.
I realise there are ways we could address many of the points I've raised here, e.g., reindexing data, renaming fields in the Logstash pipeline, etc, however when you consider everything that would be required to pull this off, it's a massive amount of very risky work, just to get around a mapping change.
I do have a suggestion: If Beats could be optionally configured with a custom root object for host metadata, this would make our problems disappear entirely. As per my previous comment, if we could use something like
instance this would suit us just fine, as its descriptive enough, and we obviously don't have any pre-existing mappings for host metadata to worry about.