Hello,
we have registered an agent running on AWS ECS to collect our Kubernetes logs from AWS Cloudwatch. We are using start_position: lastSync and a persistent AWS EFS volume for the agent's state.
We are fetching two types of Kubernetes logs:
-
audit logs with Elastic's native Kubernetes audit logs via Cloudwatch integration
-
authenticator logs
For this one there is no nice integration which is why we use the standard Cloudwatch integration with custom parsing in the agent's processor
For both we are matching log groups by the common prefix /aws/eks/ and different log stream prefixes (kube-apiserver-audit- vs. authenticator-).
However, in the state file with the cursor there is only a single entry with the group prefix which now matches both inputs
{"op":"set","id":104}
{"k":"filebeat::aws-cloudwatch::state::groupPrefix::/aws/eks/test/::eu-central-1","v":{"last_sync_epoch":1781872091842}}
We now had an issue that the audit logs input was a bit behind and did not catch up, while the authenticator logs input had processed everything. The agent then restarted and the audit logs input took the last_sync_epoch from the state which was written by the authenticator logs input.
Of course, we can try to implement some dirty workarounds by using maybe /aws/eks and /aws/eks/ as prefixes. But this is not a clean solution and other people might run into this as well.