Summary: When running Filebeat's 7.8.0 setup command the field mappings for the majority of modules which define mappings of type alias never get created. The alias mappings are missing from export template output as well. Code review shows this is affecting only aliases declared with migration: true in fields.yaml.
Details: This was initially discovered while attempting to perform a first-time setup using Filebeat package v7.8.1 installed on Ubuntu 18.04 LTS from Elastic's Debian repository at https://artifacts.elastic.co/packages/7.x/apt (suite: stable/main, components: amd64, Packages) and Nginx module enabled with "filebeat modules enable nginx" using a new Elasticsearch Service on Elastic Cloud deployment (Elasticsearch 7.8) as the configured output of Filebeat. None of the alias fields listed in the Filebeat Reference 7.8, Exported fields, Nginx fields were present in the created index template.
A closer investigation of the Filebeat log output with logging level set to debug as well as the output of filebeat setup template reveals that it's not only Nginx module that is affected. The only modules that have alias mappings present are traefik and suricata with all of the others (e.g. Apache, Beat, Docker, Elasticsearch, Kafka and Redis) affected by this.
Temporary workaround: After some digging around both open and closed Beats Github issues I have come across pull request: Do not configure aliases in unsupported Elasticsearch versions #9992 which introduced a check in "libbeat/template/processor.go" to only configure aliases in Elasticsearch versions 6.4 and above. Commenting it out did resolve the issue, but commenting the conditional check "if !p.Migration && f.MigrationAlias" that followed it (commented as "In case migration is disabled and it's a migration alias, field is not created") actually did:
func (p *Processor) alias(f *mapping.Field) common.MapStr {
// Aliases were introduced in Elasticsearch 6.4, ignore if unsupported
if p.EsVersion.LessThan(common.MustNewVersion("6.4.0")) {
return nil
}
// In case migration is disabled and it's a migration alias, field is not created
//if !p.Migration && f.MigrationAlias {
// return nil
//}
properties := getDefaultProperties(f)
properties["type"] = "alias"
properties["path"] = f.AliasPath
return properties
}
Following this modification the missing aliases are now present when running filebeat setup and also in the output of filebeat export template.
Here's part of the filebeat export template output for the Nginx module before the modification:
{
"nginx": {
"properties": {
"access": {
"properties": {
"geoip": {
"properties": {}
},
"user_agent": {
"properties": {}
}
}
},
"ingress_controller": {
"properties": {
"upstream": {
"properties": {
"alternative_name": {
"ignore_above": 1024,
"type": "keyword"
},
"port": {
"type": "long"
},
"response": {
"properties": {
"status_code": {
"type": "long"
},
"length": {
"type": "long"
},
"time": {
"type": "double"
}
}
},
"ip": {
"type": "ip"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"geoip": {
"properties": {}
},
"http": {
"properties": {
"request": {
"properties": {
"length": {
"type": "long"
},
"time": {
"type": "double"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
"user_agent": {
"properties": {}
}
}
},
"error": {
"properties": {
"connection_id": {
"type": "long"
}
}
}
}
}
}
And after the modification:
{
"nginx": {
"properties": {
"access": {
"properties": {
"referrer": {
"path": "http.request.referrer",
"type": "alias"
},
"response_code": {
"path": "http.response.status_code",
"type": "alias"
},
"agent": {
"path": "user_agent.original",
"type": "alias"
},
"geoip": {
"properties": {
"continent_name": {
"path": "source.geo.continent_name",
"type": "alias"
},
"region_iso_code": {
"path": "source.geo.region_iso_code",
"type": "alias"
},
"city_name": {
"path": "source.geo.city_name",
"type": "alias"
},
"country_iso_code": {
"path": "source.geo.country_iso_code",
"type": "alias"
},
"location": {
"path": "source.geo.location",
"type": "alias"
},
"region_name": {
"path": "source.geo.region_name",
"type": "alias"
}
}
},
"method": {
"path": "http.request.method",
"type": "alias"
},
"user_name": {
"path": "user.name",
"type": "alias"
},
"http_version": {
"path": "http.version",
"type": "alias"
},
"body_sent": {
"properties": {
"bytes": {
"path": "http.response.body.bytes",
"type": "alias"
}
}
},
"url": {
"path": "url.original",
"type": "alias"
},
"user_agent": {
"properties": {
"original": {
"path": "user_agent.original",
"type": "alias"
},
"os": {
"path": "user_agent.os.full_name",
"type": "alias"
},
"name": {
"path": "user_agent.name",
"type": "alias"
},
"os_name": {
"path": "user_agent.os.name",
"type": "alias"
},
"device": {
"path": "user_agent.device.name",
"type": "alias"
}
}
}
}
},
"ingress_controller": {
"properties": {
"referrer": {
"path": "http.request.referrer",
"type": "alias"
},
"agent": {
"path": "user_agent.original",
"type": "alias"
},
"response_code": {
"path": "http.response.status_code",
"type": "alias"
},
"upstream": {
"properties": {
"alternative_name": {
"ignore_above": 1024,
"type": "keyword"
},
"port": {
"type": "long"
},
"response": {
"properties": {
"status_code": {
"type": "long"
},
"length": {
"type": "long"
},
"time": {
"type": "double"
}
}
},
"ip": {
"type": "ip"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"geoip": {
"properties": {
"region_iso_code": {
"path": "source.geo.region_iso_code",
"type": "alias"
},
"continent_name": {
"path": "source.geo.continent_name",
"type": "alias"
},
"city_name": {
"path": "source.geo.city_name",
"type": "alias"
},
"country_iso_code": {
"path": "source.geo.country_iso_code",
"type": "alias"
},
"location": {
"path": "source.geo.location",
"type": "alias"
},
"region_name": {
"path": "source.geo.region_name",
"type": "alias"
}
}
},
"method": {
"path": "http.request.method",
"type": "alias"
},
"user_name": {
"path": "user.name",
"type": "alias"
},
"http": {
"properties": {
"request": {
"properties": {
"length": {
"type": "long"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"time": {
"type": "double"
}
}
}
}
},
"http_version": {
"path": "http.version",
"type": "alias"
},
"body_sent": {
"properties": {
"bytes": {
"path": "http.response.body.bytes",
"type": "alias"
}
}
},
"user_agent": {
"properties": {
"original": {
"path": "user_agent.original",
"type": "alias"
},
"os": {
"path": "user_agent.os.full_name",
"type": "alias"
},
"name": {
"path": "user_agent.name",
"type": "alias"
},
"os_name": {
"path": "user_agent.os.name",
"type": "alias"
},
"device": {
"path": "user_agent.device.name",
"type": "alias"
}
}
},
"url": {
"path": "url.original",
"type": "alias"
}
}
},
"error": {
"properties": {
"connection_id": {
"type": "long"
},
"level": {
"path": "log.level",
"type": "alias"
},
"pid": {
"path": "process.pid",
"type": "alias"
},
"message": {
"path": "message",
"type": "alias"
},
"tid": {
"path": "process.thread.id",
"type": "alias"
}
}
}
}
}
}
I went back through all Filebeat releases, downloading and checking output of filebeat export tempalte for versions 7.0, 7.1 etc. to version 7.7 and all exhibited the same behavior. The last version where this was not the case was version 6.8 where fields.yaml did not specify the types of the fields as aliases yet.
Other users seem to have encountered this, e.g. user @prophoto was searching for Apache field names but failed to find any in Kibana's Discover, see the thread 7.8 doesnāt parse apache access logs.