Hi,
I have a very strange issue I can't pinpoint.
add_kubernetes_metadata works - I see the fields
rename works - I can rename fields
BUT, I can't rename the kubernetes fields. I added the -d * switch to filebeat to debug the issue, and this is the error:
| 2018-07-16T13:52:55.629Z | DEBUG | [rename] | actions/rename.go:64 | Failed to rename fields, revert to old event: could not fetch value for key: metadata.container, Error: key not found |
|---|---|---|---|---|
| 2018-07-16T13:52:55.629Z | DEBUG | [filter] | pipeline/processor.go:157 | fail to apply processor client{add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]], add_kubernetes_metadata, rename=[{From:kubernetes.pod.name To:metadata.container}]}: could not fetch value for key: metadata.container, Error: key not found |
Here's the full example:
I add docker and kubernetes metadata, I see the fields both in filebeat's console and in Kibana:
processors:
- add_docker_metadata: ~
- add_kubernetes_metadata:
in_cluster: true
In particular, I see the kubernetes.pod.name field. Now I just add rename:
processors:
- add_docker_metadata: ~
- add_kubernetes_metadata:
in_cluster: true
- rename:
fields:
- from: "kubernetes.pod.name"
to: "metadata.container"
And it doesn't work. The errors, as written above:
| 2018-07-16T14:11:54.026Z | DEBUG | [rename] | actions/rename.go:64 | Failed to rename fields, revert to old event: could not fetch value for key: metadata.container, Error: key not found |
|---|---|---|---|---|
| 2018-07-16T14:11:54.026Z | DEBUG | [filter] | pipeline/processor.go:157 | fail to apply processor client{add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]], add_kubernetes_metadata, rename=[{From:kubernetes.pod.name To:metadata.container}]}: could not fetch value for key: metadata.container, Error: key not found |
And in the published event, "kubernetes.pod" is completely emptied out, as if it's deleted.
Notice it says "could not fetch value for key: metadata.container" but why would it want to find the target key?
I went over rename.go (https://github.com/ruflin/beats/blob/af065c8bfd7931a7f3ba946360946a891ac26cbd/libbeat/processors/actions/rename.go) and found at least one bug:
In line 86,
should be:
return fmt.Errorf("could not fetch value for key: %s, Error: %s", from, err)
instead of:
return fmt.Errorf("could not fetch value for key: %s, Error: %s", to, err)
as we are searching for the source, not the target, but that's just a bug in the error message.
I still don't understand why would fields.GetValue (line 80) would return an error if the key exists. It's like somehow the renaming is causing a deletion or something.
Any ideas??? A workaround would be greatly appreciated!
Thanks in advance,
Tal
). Just wanting to make sure we don't miss anything. Especially thinking could it be related to the internal queue or something like that.
One additional thing we could try: