I'm unable to rename sub-fields of the cloud field added via add_cloud_metadata.
In particular, this configuration results in missing fields for serverId.id, serverId.datacenter.zone, etc:
- add_cloud_metadata: ~
- rename:
fields:
- from: "cloud.instance.id"
to: "serverId.id"
- from: "cloud.availability_zone
to: "serverId.datacenter.zone"
- from: "cloud.region"
to: "serverId.datacenter.region"
- from: "cloud.provider"
to: "serverId.datacenter.provider"
ignore_missing: true
What's even more odd is that the original fields under cloud are removed from the output. In the case of cloud.instance.id, we're left with cloud.instance: {}.
If we remove the ignore_missing flag, we end up with an error about missing fields. What's odd is that drop_fields appears to work fine and can drop sub-fields of cloud. Also, renaming the top-level cloud field works fine.
For example, this works:
- add_cloud_metadata: ~
- drop_fields:
fields: ["cloud.image", "cloud.machine", "cloud.account"]
- rename:
fields:
- from: "cloud"
to: "serverId.cloud"
I'm using journalbeat 7.3.2, but I don't believe this issue is specific to journalbeat.
I'm also able to rename sub-fields of the host field added by add_host_metadata just fine. This issue appears to be specific to add_cloud_metadata.
Comparing https://github.com/elastic/beats/blob/master/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go to https://github.com/elastic/beats/blob/master/libbeat/processors/add_host_metadata/add_host_metadata.go, could the issue be with the lack of cloning the underlying metadata fields?